Review

# The existence of positive solutions to an elliptic system with nonlinear boundary conditions

Chunhua Wang* and Jing Yang

### Author affiliations

School of Mathematics and Statistics, Huazhong Normal University, Wuhan, 430079, P.R. China

For all author emails, please log on.

Boundary Value Problems 2013, 2013:159  doi:10.1186/1687-2770-2013-159

 Received: 8 April 2013 Accepted: 16 June 2013 Published: 1 July 2013

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

### Abstract

In this paper, we consider the following system:

{ Δ u = u , Δ v = v , x Ω , u ν = f ( x , v ) , v ν = g ( x , u ) , x Ω ,

where Ω is a bounded domain in R N ( N 3 ) with smooth boundary, ν is the outer normal derivative and f , g : Ω × R R + are positive and continuous functions. Under certain assumptions on f ( x , t ) and g ( x , t ) , but without the usual (AR) condition, we prove that the problem has at least one positive strong pair solution ( u , v ) (see Definition 1.4 below) by applying a linking theorem for strong indefinite functional.

MSC: 35A01, 35J20, 35J25.

##### Keywords:
fractional Sobolev spaces; linking theorem; nonlinear boundary conditions; strong solution

### 1 Introduction and main result

In this paper, we mainly study the following system:

{ Δ u = u , Δ v = v , x Ω , u ν = f ( x , v ) , v ν = g ( x , u ) , x Ω , (1.1)

where Ω is a bounded domain in R N ( N 3 ) with smooth boundary, ν is the outer normal derivative and f , g : Ω × R R + are positive and continuous functions.

Existence results for nonlinear elliptic systems have received a lot of interest in recent years (see [1-12]), particularly when the nonlinear term appears as a source in the equation, complemented with Dirichlet boundary conditions. To our knowledge, about the system with nonlinear boundary conditions, there are not many results. Here we refer to [9,13,14].

We are mainly motivated by [12] and [14].

In [12], Li and one of the authors considered

{ Δ u + u = f ( x , v ) , x R N , Δ v + v = g ( x , u ) , x R N . (1.2)

Under some given conditions, we proved that (1.2) had at least one positive solution pair ( u , v ) H 1 ( R N ) × H 1 ( R N ) .

In [14], Bonder, Pinasco, Rossi studied

{ Δ u = u , Δ v = v , x Ω , u ν = H v ( x , u , v ) , v ν = H u ( x , u , v ) , x Ω . (1.3)

They assumed that H satisfied the following conditions:

( H ˜ 1 ) | H ( x , u , v ) | C ( | u | p + 1 + | v | q + 1 + 1 ) .

( H ˜ 2 ) The Ambrosetti-Rabinowitz type condition: For R large, if | ( u , v ) | R ,

1 α H u ( x , u , v ) u + 1 β H v ( x , u , v ) u H ( x , u , v ) > 0 , (1.4)

where p + 1 α > p > 0 and q + 1 β > q > 0 with

1 > 1 α + 1 β , (1.5)

max { p α + q β ; q q + 1 p + 1 α + p p + 1 q + 1 β } < 1 + 1 N 1 , (1.6)

q q + 1 p + 1 α < 1 , and p p + 1 q + 1 β < 1 . (1.7)

When N 4 , they also assumed

max { p α + q β ; q q + 1 p + 1 α + p p + 1 q + 1 β } < N + 1 2 ( N 1 ) . (1.8)

( H ˜ 3 ) | H u ( x , u , v ) | ( | u | p + | v | p ( q + 1 ) p + 1 + 1 ) , | H v ( x , u , v ) | ( | u | q ( p + 1 ) q + 1 + | v | q + 1 ) .

( H ˜ 4 ) H ( x , u , v ) = H ( x , u , v ) .

They obtained infinitely many nontrivial solutions of (1.3) under the assumptions ( H ˜ 1 ) to ( H ˜ 4 ) by using variational arguments and a fountain theorem. Note that (1.4) implies

| H ( x , u , v ) | c ( | u | α + | v | β ) C (1.9)

(see Lemma 1.1 in [6]). Therefore it is not difficult to verify that any (PS) sequence (or (C)c sequence) of the corresponding functional is bounded in some suitable space.

The crucial part in the nonlinear boundary conditions case is to find the proper functional setting for (1.1) that allows us to treat our problem variationally. We accomplish this by defining a self-adjoint operator that takes into account the boundary conditions together with the equations and considering its fractional powers that satisfy a suitable ‘integration by parts’ formula. In order to obtain nontrivial solutions, we use a linking theorem (see [11]).

The assumptions we impose on f ( x , t ) and g ( x , t ) are as follows:

(H1) f , g C 0 ( Ω × R 1 , R 1 ) with f ( x , t ) = g ( x , t ) = 0 for any ( x , t ) Ω × ( , 0 ] ,

f ( x , t ) > 0 and g ( x , t ) > 0 for any  ( x , t ) Ω × ( 0 , + ) .

(H2) lim t 0 f ( x , t ) t = lim t 0 g ( x , t ) t = 0 uniformly in x Ω .

(H3) There is a positive constant C > 0 such that

| f ( x , t ) | C ( 1 + | t | p 1 ) , | g ( x , t ) | C ( 1 + | t | q 1 ) , ( x , t ) Ω × R 1 , (1.10)

where p , q > 2 and satisfy

1 p + 1 q > 1 1 N 1 . (1.11)

(H4) lim | t | + F ( x , t ) t 2 = lim | t | + G ( x , t ) t 2 = + uniformly in x Ω , where F ( x , t ) : = 0 t f ( x , s ) d s , G ( x , t ) : = 0 t g ( x , s ) d s .

(H5) For all 0 < t < s , x Ω or s < t < 0 , x Ω , there are two positive constants C 1 , , C 2 , such that

H 1 ( x , t ) H 1 ( x , s ) + C 1 , , H 2 ( x , t ) H 2 ( x , s ) + C 2 , , (1.12)

where H 1 ( x , t ) = t f ( x , t ) 2 F ( x , t ) , H 2 ( x , t ) = t g ( x , t ) 2 G ( x , t ) with H 1 ( x , t ) , H 2 ( x , t ) > 0 for any t > 0 , x Ω .

Remark 1.1 By (1.11), there exist l and m with l + m = 1 , l , m > 1 4 such that

1 p > 1 2 2 l 1 2 N 1 , 1 q > 1 2 2 m 1 2 N 1 . (1.13)

Remark 1.2 (H5) was first introduced by Miyagaki and Souto in [15]. A typical pair of functions f ( x , t ) = t p 1 , g ( x , t ) = t q 1 , t > 0 , p , q > 2 ; f ( x , t ) = g ( x , t ) = 0 , t 0 satisfy (H1) to (H5). However, the pair of functions f ( x , t ) = t ( 2 ln t + 1 ) , g ( x , t ) = t ( 2 ln t + 1 ) , t > 0 ; f ( x , t ) = g ( x , t ) = 0 , t 0 satisfy (H5) but do not satisfy the usual (AR) condition and ( H ˜ 2 ) in this paper.

Remark 1.3 The assumptions we impose on f and g are different from the assumptions in [14]. To our best knowledge, it is the first time the group assumptions have been used to deal with a system with nonlinear boundary conditions.

In order to state our main result, first we give a definition.

Definition 1.4 We say that ( u , v ) is a strong solution of (1.1) if

u W 2 , p p 1 ( Ω ) , v W 2 , q q 1 ( Ω )

and ( u , v ) satisfies (1.1) a.e. in Ω.

Our main results is as follows.

Theorem 1.5Let (H1)-(H5) hold. Then system (1.1) possesses at least one positive strong solution pair z = ( u , v ) .

The main difficulties to deal with system (1.1) consist in at least three aspects. Firstly, due to the type of growth of the functions f and g, we cannot work with the usual H 1 ( Ω ) , and then we need fractional Sobolev spaces. Secondly, although we have a variational problem, the functional associated to it always has a strong indefinite quadratic part. So, the functional possesses no mountain-pass structure but the linking geometric structure, which is more complicated to handle. Thirdly, as we do not assume that the functions f and g satisfy the (AR) conditions, it is much more difficult to show that any (C)c sequence is uniformly bounded in E (see Section 2).

To prove Theorem 1.5, we try to find a critical point of the functional Φ (see (2.5)) in E. We prove that Φ has a linking geometric structure and use a linking theorem under (C)c condition (see Theorem 2.1 in [11]) to get a (C)c-sequence { z n } E of Φ. The main difficulty now will be to prove that { z n } is uniformly bounded in E without the (AR) condition. Then we prove that any (C)c-sequence { z n } E of Φ is bounded. To overcome this difficulty, we use some techniques used in [12,16] for which the assumptions (H4), (H5) play important roles. As { z n } is bounded, then we can prove that { z n } has a subsequence which converges to a nontrivial critical point of Φ. Hence, by the strong maximum principle, we can prove that the pair solution ( u , v ) is positive.

The paper is organized as follows. In Section 2, we give some preliminaries. We prove our main result in Section 3.

### 2 Some preliminaries

In this section we mainly give some preliminaries which will be used in Section 3. We follow the structure in [13].

Throughout this paper, we consider the space L 2 ( Ω ) × L 2 ( Ω ) which is a Hilbert space with the inner product, which we denote by , , given by

( u , v ) , ( φ , ψ ) = Ω u φ d x + Ω v ψ d σ , for any  ( u , v ) , ( φ , ψ ) L 2 ( Ω ) × L 2 ( Ω ) .

Now we let A : D ( A ) L 2 ( Ω ) × L 2 ( Ω ) L 2 ( Ω ) × L 2 ( Ω ) be the operator defined by

A ( u , u | Ω ) = ( Δ u + u , u ν ) ,

where D ( A ) = { ( u , u | Ω ) : u H 2 ( Ω ) } . It is not difficult to verify that D ( A ) is dense in L 2 ( Ω ) × L 2 ( Ω ) . Note that A is invertible with its inverse given by

A 1 ( h 1 , h 2 ) = ( u , u | Ω ) ,

where u is the solution of

{ Δ u + u = h 1 , in  Ω , u ν = h 2 , on  Ω . (2.1)

By standard regularity (see [17]), it follows that A 1 is bounded and compact. Hence, R ( A ) = L 2 ( Ω ) × L 2 ( Ω ) . Therefore, in order to see that A (hence A 1 ) is self-adjoint, it suffices to prove that A is symmetric ([18], p.512). In fact, for u , v D ( A ) , applying Green’s formula, we obtain

A u , v = Ω ( Δ u + u ) v d x + Ω u ν v d σ = Ω ( Δ v + v ) u d x + Ω v ν u d σ = v , A u .

Hence A is symmetric. Also we can check that A (and so A 1 ) is positive. For any u D ( A ) and by Green’s formula again, we have

A u , u = Ω ( Δ u + u ) u d x + Ω u ν u d σ = Ω ( | u | 2 + u 2 ) d x 0 .

Hence there is a sequence of eigenvalues ( λ n ) R with eigenfunctions ( φ n , ψ n ) L 2 ( Ω ) × L 2 ( Ω ) satisfying 0 < λ 1 λ 2 λ n and φ n H 2 ( Ω ) , φ n | Ω = ψ n ,

{ Δ φ n + φ n = λ n φ n , in  Ω , φ n ν = λ n ψ n , on  Ω . (2.2)

Now we consider the following fractional powers of A, i.e., for 0 < l < 1 ,

A l : D ( A l ) L 2 ( Ω ) × L 2 ( Ω ) , with  A l u = n = 1 λ n l a n ( φ n , ψ n ) ,

where u = n = 1 a n ( φ n , ψ n ) . Let E l = D ( A l ) , which is a Hilbert space under the inner product

( u , φ ) E l = A l u , A l φ .

Note that E l H 2 l ( Ω ) . Indeed, if we define A 1 : H 2 ( Ω ) L 2 ( Ω ) L 2 ( Ω ) by

A 1 u = Δ u + u ,

and A 2 : H 2 ( Ω ) D ( A 2 ) L 2 ( Ω ) L 2 ( Ω ) by

A 2 u = u ν ,

then A ¯ = ( A 1 , A 2 ) satisfies

A = A ¯ | ( u , u ) , u D ( A 1 ) D ( A 2 ) ,

and hence

A l = A ¯ l | ( u , u ) , u D ( A 1 l ) D ( A 2 l ) .

Since D ( A 1 ) = H 2 ( Ω ) D ( A 2 ) , we have D ( A 1 l ) D ( A 2 l ) . Therefore,

E l = D ( A l ) = D ( A 1 l ) .

Noting that Ω is smooth, it follows from the results of p.187 in [19] (see also [18,20]) that

E l = D ( A 1 l ) H 2 l ( Ω ) .

The following compact result will be useful later.

Proposition 2.1 (Theorem 2.1, [14])

Given l > 1 4 and r 1 so that 1 r 1 2 2 l 1 2 N 1 , the inclusion map i : E l L r ( Ω ) is well defined and bounded. Moreover, if 1 r > 1 2 2 l 1 2 N 1 , then the inclusion is compact.

Denote E = E l × E m , where l + m = 1 , l, m are the same as in Remark 1.1 and define B : E × E R by

B ( ( u , v ) , ( φ , ψ ) ) = A l u , A m ψ + A l φ , A m v .

Associated to B, we have the quadratic form

Q ( z ) : = 1 2 B ( z , z ) = A l u , A m v .

It is easy to see (one can refer to [14]) that the bounded self-adjoint operator L : E E defined by ( L z , η ) : = B ( z , η ) has exactly two eigenvalues +1 and −1, and that the corresponding eigenvalues E + and E are given by

E + = { ( u , A m A l u ) : u E l } and E = { ( u , A m A l u ) : u E l } ,

where we use the notation A m = ( A m ) 1 . Then E = E + E . The spaces E + and E are orthogonal with respect to the bilinear B, that is,

B ( z + , z ) = 0 , z + E + , z E .

Moreover, we have

1 2 z ± E 2 = Q ( z ± ) = Ω | A l u | 2 d x ,

if z ± = ( u , ± ( A m A l u ) ) . We see also that for z = ( u , v ) , z = z + + z with z + = ( ( u + A l A m v ) 2 , ( A m A l u + v ) 2 ) , z = ( ( u A l A m u v ) 2 , ( A m A l u + v ) 2 ) and

A l u , A m v = 1 2 B ( z , z ) = 1 2 L z , z = 1 2 ( z + E 2 z E 2 ) . (2.3)

From (1.10), Remark 1.1 and Proposition 2.1, we can define the functional H : E R as

H ( u , v ) = Ω F ( x , v ) d σ + Ω G ( x , u ) d σ . (2.4)

Lemma 2.2The functionaldefined by (2.4) is of class C 1 and its derivative is given by

H ( u , v ) ( φ , ψ ) = Ω f ( x , v ) ψ d σ + Ω g ( x , u ) φ d σ .

Moreover, H is compact.

Proof From (1.10), Hölder’s inequality and Proposition 2.1, we have

| Ω f ( x , v ) ψ d σ | Ω | f ( x , v ) ψ | d σ Ω ( 1 + | v | p 1 ) | ψ | d σ v L p ( Ω ) p 1 ψ L p ( Ω ) + ψ L p ( Ω ) | Ω | p 1 p C ( v E m p 1 + 1 ) ψ E m .

Similarly, we have

| Ω g ( x , u ) φ d σ | C ( u E l q 1 + 1 ) φ E l .

Hence H is well defined and bounded in E. A standard argument yields that ℋ is Fréchet differentiable with H continuous. By Proposition 2.1 we know that H is compact (see [21] for the details). □

Now we define the functional Φ : E R for (1.1) given by

Φ ( z ) = Q ( z ) H ( z ) . (2.5)

Moreover, Φ is class C 1 ( E , R ) .

Definition 2.3 We say that z = ( u , v ) E = E l × E m is an ( l , m ) -weak solution of (1.1) if z is a critical point of Φ. In other words, for every ( φ , ψ ) E , we have

A l u , A m ψ + A l φ , A m v Ω f ( x , v ) ψ d σ Ω g ( x , u ) φ d σ = 0 . (2.6)

Now we give a regularity result of an ( l , m ) weak solution.

Proposition 2.4If ( u , v ) E is an ( l , m ) -weak solution of (1.1), then u W 2 , p p 1 ( Ω ) , v W 2 , q q 1 ( Ω ) and

Δ u = u in   Ω , u ν = f ( x , v ) on   Ω , (2.7)

Δ v = v in   Ω , v ν = g ( x , u ) on   Ω . (2.8)

In other words, ( u , v ) is a strong solution of (1.1).

Proof Although the proof is only needed to make some minor modifications as that of Theorem 2.2 in [13], for the readers’ convenience, we give its detailed proof.

Let us consider φ = 0 in (2.6), then

A l u , A m ψ Ω f ( x , v ) ψ = 0 (2.9)

for all ψ E m .

If we take ψ H 2 ( Ω ) , then we have

A l u , A m ψ = u , A ψ = Ω ( Δ ψ + ψ ) u d x + Ω ψ ν u d σ . (2.10)

On the other hand, by (1.10) and Proposition 2.1, we have

Ω | f ( x , v ) | p p 1 d σ Ω ( 1 + | v | p 1 ) p p 1 d σ C Ω ( 1 + | v | p ) d σ C + C v E m p < + , (2.11)

i.e., f ( x , v ) L p p 1 ( Ω ) . Then from basic elliptic theory (Theorem 9.9, p.9, [17]) there exists one function w W 2 , p p 1 ( Ω ) such that

Δ w = w in  Ω , w ν = f ( x , v ) on  Ω .

Then we get

0 = Ω ( Δ w + w ) ψ d x = Ω ( Δ ψ + ψ ) w d x + Ω w ψ ν d σ Ω f ( x , v ) ψ d σ . (2.12)

From (2.10), (2.11) and (2.12), we have

u w , A ψ = Ω ( u w ) ( Δ ψ + ψ ) d x + Ω ( u w ) ψ ν = 0 ,

which implies that u = w . We have gotten that u W 2 , p p 1 ( Ω ) . Finally, since u = w , we conclude that u satisfies (2.7). We can make the same argument for v. □

### 3 The proof of our main result

In this section, we mainly want to prove Theorem 1.5. First we present a linking theorem from [11]. Then we prove that it can be applied to our functional setting stated in Section 2.

Suppose that f ( x , t ) , g ( x , t ) satisfy the assumptions (H1)-(H3), then it is easy to see that for any ϵ > 0 there is a C ϵ > 0 such that for ( x , t ) Ω × R 1 we have

| f ( x , t ) | ϵ | t | + C ϵ | t | p 1 , | g ( x , t ) | ϵ | t | + C ϵ | t | q 1 (3.1)

and

| F ( x , t ) | ϵ | t | 2 + C ϵ | t | p , | G ( x , t ) | ϵ | t | 2 + C ϵ | t | q . (3.2)

Since f ( x , t ) = g ( x , t ) = 0 when t 0 , ( u , v ) = ( 0 , 0 ) E is a solution of (1.1). So we are interested in nontrivial and nonnegative solutions of (1.1).

Recall that { z n } E is called a Palais-Smale sequence of a C 1 functional I on E at level c ((PS)c-sequence for short) if I ( z n ) c and I ( z n ) 0 in E as n . If I ( z n ) c and ( 1 + z n E ) I ( z n ) 0 in E as n , then { z n } will be called a Cerami sequence at level c ((C)c-sequence for short). A standard way to prove the existence of a positive solution to (1.1) is to get a (PS)c or (C)c sequence for Φ and then to prove that the sequence converges to a solution to (1.1). In this paper, we want to get a (C)c sequence by a linking theorem (Theorem 2.1, in [11]). So, we need to recall some terminology (see, e.g., [11,22]).

Let H be a closed separable subspace of a Hilbert space H with the norm H and let H + : = ( H ) . For u H , we shall write u = u + + u , where u ± H ± . On H we define a new norm

u τ : = max { u + H , k = 1 1 2 k | u , e k | } ,

where { e k } is a total orthonormal sequence in H . The topology generated by τ will be called the τ-topology. Recall from [22] that a homotopy h = I g : A × [ 0 , 1 ] H , where A H , is called admissible if:

(i) h is τ-continuous, i.e., h ( u n , s n ) h ( u , s ) in τ-topology as n whenever u n u in τ-topology and s n s as n .

(ii) g is τ-locally finite-dimensional, i.e., for each ( u , s ) A × [ 0 , 1 ] , there is a neighborhood U of ( u , s ) in the product topology of ( E , τ ) and [ 0 , 1 ] such that g ( U ( A × [ 0 , 1 ] ) ) is contained in a finite-dimensional subspace of H.

Admissible maps are defined similarly. Recall also that admissible maps and homotopies are necessarily continuous, and on bounded subsets of H the τ-topology coincides with the product topology of H weak and H strong + .

Let Φ C 1 ( H , R 1 ) , R > r > 0 and z 0 H + { 0 } and define

M = { z = z + t z 0 : z R , t 0 } , N = { z H + : z = r }

and

Γ : = { h C ( M × [ 0 , 1 ] , H ) | h  is admissible , h ( u , 0 ) = u  and Φ ( h ( u , s ) ) max { Φ ( u ) , 1 }  for all  s [ 0 , 1 ] } .

Proposition 3.1 (Theorem 2.1, [11])

Let H = H + H be a separable Hilbert space with H orthogonal to H + . Suppose that

(i) Φ ( z ) = 1 2 ( z + z 2 ) Ψ ( z ) , where Ψ C 1 ( H , R 1 ) is bounded below, weakly sequentially lower semi-continuous and Ψ is weakly sequentially continuous.

(ii) There exist z 0 H + { 0 } , α > 0 , and R > r > 0 such that Φ | N α and Φ | M 0 .

Then there exists a (C)c-sequence for Φ, where

c : = inf h Γ sup u M Φ ( h ( u , 1 ) ) .

Moreover, c α .

For fixed z 0 E + { 0 } and R > r > 0 , let

M R = { z = z + ρ z 0 : z E , z E R , ρ 0 } , N r = { z E + : z E = r } .

Lemma 3.2There exist r > 0 and α > 0 such that Φ | N r α .

Proof For any z N r , z = ( u , v ) E + , we know that v = A m A l u or, equivalently, u = A l A m v . By (3.2) and Proposition 2.1, we have

| Ω F ( x , v ) d σ + Ω G ( x , u ) d σ | Ω ( | F ( x , v ) | + | G ( x , u ) | ) d σ Ω [ ( ϵ | v | 2 + C ϵ | v | p ) + ( ϵ | u | 2 + C ϵ | u q ) ] d σ ϵ u E l 2 + C ϵ u E l p + ϵ v E m 2 + C ϵ v E m q ϵ z E 2 + C ϵ z E p + ϵ z E 2 + C ϵ z E q .

Since p , q > 2 , we have if z E = r is small enough,

Φ ( z ) = Ω A l u A m v d x Ω F ( x , v ) d σ Ω G ( x , u ) d σ 1 2 z E 2 ϵ z E 2 C ϵ z E p C ϵ z E q α > 0

for some α > 0 . □

Lemma 3.3For thergiven by Lemma 3.2 and any z 0 = ( u 0 , v 0 ) E + { 0 } with z 0 E = 1 , there exists R > r such that Φ | M R 0 , where M R = { z = z + ρ z 0 : z E R , ρ 0 } .

Proof If z M R , then z = z + ρ z 0 with either : z E = R , ρ 0 or : z E < R , ρ = 0 .

(i) If ρ = 0 , then we have z E , z = ( u , A m A l u ) and

Φ ( z ) = 1 2 z E 2 Ω F ( x , v ) Ω G ( x , u ) 0 ,

since F ( x , t ) , G ( x , t ) 0 for any ( x , t ) Ω × R 1 .

(ii) Assume that ρ > 0 . We argue by contradiction. Suppose that there exists a sequence { z n } M n , z n = ρ n z 0 + z n , ρ n > 0 , z 0 E = 1 , z n E = n such that Φ ( z n ) > 0 . If z n = ( u n , v n ) : = ( ρ n u 0 + φ n , ρ n v 0 + ψ n ) , then by the definitions of E + and E , we have

A l u n , A m v n = Ω A l u n A m v n d x = Ω A l ( ρ n u 0 + φ n ) A m ( ρ n v 0 + ψ n ) d x = Ω ( ρ n A l u 0 + A l φ n ) ( ρ n A m v 0 + A m ψ n ) d x = Ω ( ρ n 2 | A l u 0 | 2 ρ n A l u 0 A l φ n + ρ n A l u 0 A l φ n | A m ψ n | 2 ) d x = ρ n 2 2 z 0 E 2 1 2 z n E 2 .

Hence,

Φ ( z n ) = 1 2 ( ρ n 2 z 0 E 2 z n E 2 ) Ω F ( x , v n ) d σ Ω G ( x , u n ) d σ > 0 .

Therefore,

Φ ( z n ) z n E 2 = 1 2 ( ρ n 2 z n E 2 z 0 E 2 z n E 2 z n E 2 ) Ω F ( x , v n ) + G ( x , u n ) z n E 2 d σ > 0 .

Denote δ n : = ρ n z n E , w n : = z n z n E = ( φ ˜ n , ψ ˜ n ) . Then

Φ ( z n ) z n E 2 = 1 2 ( δ n 2 w n E 2 ) Ω F ( x , v n ) + G ( x , u n ) z n E 2 d σ > 0 . (3.3)

Since F ( x , t ) , G ( x , t ) 0 for any ( x , t ) Ω × R 1 , by (3.3) we know that δ n w n E .

On the other hand, δ n 2 + w n E 2 = 1 , which implies that δ n 2 δ 2 0 for some δ 0 and w n w = ( φ ˜ , ψ ˜ ) E as n , where ⇀ denotes the weak convergence in E.

If δ = 0 , then from (3.3) we get

w n 2 0 , Ω F ( x , v n ) z n E 2 d σ 0 , Ω G ( x , u n ) z n E 2 d σ 0 .

Therefore,

1 = δ n 2 + w n E 2 0 ,

which is impossible.

If δ > 0 , since δ n 2 δ 2 > 0 and z n E + as n , it follows that ρ n + . If x Ω is such that δ u 0 + φ ˜ 0 , we have

lim n + ρ n u 0 + φ n z n E = δ u 0 + φ ˜ ( x ) 0 ,

thus,

u n = ρ n u 0 + φ n , (3.4)

as n .

Similarly, if δ u 0 ( x ) + ψ ˜ ( x ) 0 , we have

v n = ρ n v 0 + ψ n , (3.5)

as n .

Since Φ ( z n ) z n E 2 > 0 and F ( x , t ) , G ( x , t ) 0 , we get

0 < 1 2 ( δ n 2 w n E 2 ) Ω [ F ( x , v n ) v n 2 ( v n z n E ) 2 + G ( x , u n ) u n 2 ( u n z n E ) 2 ] d σ 1 2 ( δ n 2 w n E 2 ) { δ u 0 + ψ ˜ 0 } F ( x , v n ) v n 2 ( v n z n E ) 2 d σ { δ u 0 + φ ˜ 0 } G ( x , u n ) u n 2 ( u n z n E ) 2 d σ .

Note that

u n z n E = ρ n u 0 + φ n z n E δ u 0 + φ ˜ in  E l

and

v n z n E = ρ n v 0 + ψ n z n E δ v 0 + ψ ˜ in  E m ,

as n + . Hence, by Proposition 2.1 we may assume, passing to a subsequence, that

u n z n E = ρ n u 0 + φ n z n E δ u 0 + φ ˜ , v n z n E = ρ n v 0 + ψ n z n E δ v 0 + ψ ˜ a.e. in  Ω ,

as n + . By (3.8), (3.9) and (H4), taking limit in (3.10), using Fatou’s lemma and the fact that lim inf n w n E 2 w E 2 , we obtain

0 1 2 ( δ 2 w E 2 ) { δ v 0 + ψ ˜ 0 } ( + ) ( δ v 0 + ψ ˜ ) 2 d σ { δ u 0 + φ ˜ 0 } ( + ) ( δ u 0 + φ ˜ ) 2 d σ ,

which is impossible, thus the lemma is proved. □

Lemma 3.4If { z n } is a (C)c-sequence of Φ, then { z n } is bounded inE.

Proof Suppose that { z n } E is a (C)c sequence for Φ, that is,

Φ ( z n ) c , Φ ( z n ) E ( 1 + z n E ) 0 ,

which shows that

c + o ( 1 ) = Φ ( z n ) , Φ ( z n ) , z n = o ( 1 ) , (3.6)

where o ( 1 ) 0 as n + .

z n E + , (3.7)

and let w n = z n z n E : = ( w n 1 , w n 2 ) . Then w n E with

w n E = 1 .

By Proposition 2.1, { w n } contains a subsequence, denoted again by { w n } = { ( w n 1 , w n 2 ) } such that we may assume that

{ w n 1 ( x ) w 1 ( x )  in  E l , w n 2 ( x ) w 2 ( x )  in  E m ; w n 1 ( x ) w 1 ( x ) , w n 2 ( x ) w 2 ( x )  a.e.  in  Ω ; w n 1 w  in  L α ( Ω ) ( 2 α < 2 ( N 1 ) N 4 l ) ; w n 2 w  in  L β ( Ω ) ( 2 β < 2 ( N m ) N 4 m ) . (3.8)

Let Ω = { x Ω , w ( x ) ( 0 , 0 ) } . Then we have

lim n + w n ( x ) = lim n + z n ( x ) z n E = w ( x ) ( 0 , 0 ) in  Ω ,

and (3.7) implies that

| z n | + a.e. in  Ω .

We may assume, without loss of generality, that

| u n | + a.e. in  Ω . (3.9)

By (H4), we see that

lim n + G ( x , u n ( x ) ) | u n ( x ) | 2 = + a.e. in  Ω .

This means that

lim n + G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 1 ( x ) | 2 = + a.e. in  Ω . (3.10)

By (H4), there is an N 0 > 0 such that

G ( x , s ) | s | 2 > 1 (3.11)

for any x Ω and s R 1 with | s | N 0 . Since G ( x , s ) is continuous on Ω × [ N 0 , N 0 ] , there is an M > 0 such that

| G ( x , s ) | M , (3.12)

for ( x , t ) Ω ¯ × [ N 0 , N 0 ] . From (3.11) and (3.12), we see that there is a constant C, such that for any ( x , s ) Ω × R 1 , we have

G ( x , s ) C ,

which shows that

G ( x , u n ( x ) ) C z n E 2 0 .

This means that

G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 1 ( x ) | 2 C z n E 2 0 . (3.13)

Since by (3.6) we have that

c + o ( 1 ) = Φ ( z n ) = A l u n , A m v n Ω [ F ( x , v n ) + G ( x , u n ) ] d σ ,

which shows that

0 c + o ( 1 ) z n E 2 = A l w n 1 , A m w n 2 Ω [ F ( x , v n ) + G ( x , u n ) ] z n E 2 d σ . (3.14)

Since F ( x , t ) 0 , we have

1 2 + o ( 1 ) = o ( 1 ) + 1 2 Ω ( | A l w n 1 | 2 + | A m w n 2 | 2 ) d x o ( 1 ) + A l w n 1 , A m w n 2 = Ω [ F ( x , v n ) + G ( x , u n ) ] z n E 2 d σ Ω G ( x , u n ) z n E 2 d σ . (3.15)

We claim that | Ω | = 0 .

If | Ω | 0 , then by Fatou’s lemma, (H4) and Hölder’s inequality, we get

+ = + | Ω | = [ Ω lim inf n + G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 1 ( x ) | 2 d σ Ω lim sup n + C z n E 2 d σ ] = Ω lim inf n + ( G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 1 ( x ) | 2 C z n E 2 ) d σ lim inf n + Ω ( G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 1 ( x ) | 2 C z n E 2 ) d σ lim inf n + Ω ( G ( x , u n ( x ) ) | u n ( x ) | 2 | w n 2 ( x ) | 2 C z n E 2 ) d σ = lim inf n + Ω G ( x , u n ( x ) ) u n 2 d σ lim n + Ω C z n E 2 d σ = lim inf n + Ω G ( x , u n ( x ) ) z n E 2 d σ 1 2 + o ( 1 ) ,

which is impossible.

This shows that

| Ω | = 0 .

Hence w ( x ) = ( 0 , 0 ) a.e. in Ω.

Since Φ ( t z n ) is continuous in t [ 0 , 1 ] , there exists t n [ 0 , 1 ] ( n = 1 , 2 , ), such that

Φ ( t n z n ) = max 0 t 1 Φ ( t z n ) .

As Φ ( z n ) , z n = o ( 1 ) , we see that

Φ ( t n z n ) , t n z n = o ( 1 ) .

By (H5), then we get for t [ 0 , 1 ] that

2 Φ ( t z n ) 2 Φ ( t n u n ) = 2 Φ ( t n z n ) Φ ( t n z n ) , t n z n + o ( 1 ) = Ω [ ( t n v n f ( x , t n v n ) 2 F ( x , t n v n ) ) + ( t n u n g ( x , t n u n ) 2 G ( x , t n u n ) ) ] d σ + o ( 1 ) Ω [ ( v n f ( x , v n ) 2 F ( x , v n ) + C 1 , ) + ( u n g ( x , u n ) 2 G ( x , u n ) + C 2 , ) ] d σ + o ( 1 ) = 2 c + ( C 1 , + C 2 , ) | Ω | + o ( 1 ) . (3.16)

On the other hand, taking t = R z n E [ 0 , 1 ] and z n = ( u n , u n ) , by (3.8) then w n = ( u n , u n ) z n E : = ( θ n , θ n ) E 1 × E 1 and θ n 0 in L r ( Ω ) ( 1 r < 2 ( N 1 ) N 4 ). From (3.2) and θ n 0 in L r ( Ω ) ( 1 r < 2 ( N 1 ) N 4 ) as n , we obtain

2 Φ ( R w n ) = 2 R 2 2 Ω [ F ( x , R θ n ) + G ( x , R θ n ) ] d σ = 2 R 2 + o ( 1 ) .

So we have

2 R 2 + o ( 1 ) = 2 Φ ( R w n ) 2 c + ( C 1 , + C 2 , ) | Ω | + o ( 1 ) .

Letting n , we get

2 R 2 ( C 1 , + C 2 , ) | Ω | + 2 c .

Letting R , we get a contradiction. This proves that z n E C < + for some constant C. □

Proof of Theorem 1.5 Under the assumptions (H1)-(H5), we know that the functional Φ given by (2.5) is in C 1 ( E , R 1 ) . By Lemma 3.2, there exist r > 0 and α > 0 such that Φ | N r α , where N r = { z E + : z E = r } . By Lemma 3.3, for such an r, there exist R > r and suitable z 0 E + { 0 } such that Φ | M R 0 , where M R was given before Lemma 3.3. Note that E = E + E and for z = ( u , v ) E , we have

Φ ( z ) = 1 2 z + E 2 1 2 z E 2 Ω [ F ( x , v ) + G ( x , u ) ] d σ .

Since from Proposition 2.1 and Remark 1.1 we know that E L q ( Ω ) × L p ( Ω ) , from (3.2) and Fatou’s lemma, we know that

H ( z ) = Ω [ F ( x , v ) + G ( x , u ) ] d σ

is C 1 and H 0 is weakly sequentially lower semicontinuous and H is weakly sequentially continuous in E . Hence by Proposition 3.1 there exists a (C)c-sequence { z n } for Φ, where c α > 0 . By Lemma 3.4, { z n } is bounded in E. So, up to a subsequence, we may assume that z n z ( 0 , 0 ) in E, as n . From Lemma 2.2, we know that H is compact. So it is easy to check that Φ ( z ) 0 in E . Hence z is a nontrivial solution pair of (1.1). Obviously, z = ( u , v ) E is a nonnegative solution pair of (1.1). Applying the strong maximum principle, we obtain that u > 0 and v > 0 . This completes the proof. □

### Competing interests

The authors declare that they have no competing interests.

### Authors’ contributions

JY is mainly in charge of Section 2 and CW is mainly responsible for Section 3. This paper is finished under of our joint efforts. We discuss many times and make many modifications. Both authors read and approved the final manuscript.

### Acknowledgements

The authors were partially supported by NSFC (No. 11071092; No. 11071095; No. 11101171), the PhD specialized grant of the Ministry of Education of China (20110144110001).

### References

1. Bartsch, T, de Figueiredo, DG: Infinitely many solutions of nonlinear elliptic systems. Topics in Nonlinear Analysis, pp. 51–67. Birkhäuser, Basel (1999)

2. Clément, P, de Figueiredo, DG, Mitidieri, E: Positive solutions of semilinear elliptic systems. Commun. Partial Differ. Equ.. 17, 923–940 (1992). Publisher Full Text

3. de Figueiredo, DG: Semilinear elliptic systems: a survey of superlinear problems. Resenhas. 2, 373–391 (1996)

4. de Figueiredo, DG, Felmer, PL: On superquadratic elliptic systems. Trans. Am. Math. Soc.. 343, 99–116 (1994)

5. de Figueiredo, DG, Magalhaẽs, CA: On nonquadratic Hamiltonian elliptic systems. Adv. Differ. Equ.. 1, 881–898 (1996)

6. Felmer, PL: Periodic solutions of “superquadratic” Hamiltonian systems. J. Differ. Equ.. 102, 188–207 (1993). Publisher Full Text

7. Felmer, PM, Manásevich, RF, de Thélin, F: Existence and uniqueness of positive solutions for certain quasilinear elliptic systems. Commun. Partial Differ. Equ.. 17, 1992–2029 (2013)

8. Felmer, PL, Wang, ZQ: Multiplicity for symmetric indefinite functionals: application to Hamiltonian and elliptic systems. Topol. Methods Nonlinear Anal.. 12, 207–226 (1998)

9. Fernández Bonder, J, Rossi, JD: Existence for an elliptic system with nonlinear boundary conditions via fixed point methods. Adv. Differ. Equ.. 6, 1–20 (2001)

10. Hulshof, J, van der Vorst, R: Differential systems with strongly indefinite variational structure. J. Funct. Anal.. 114, 32–58 (1993). Publisher Full Text

11. Li, GB, Szulkin, A: An asymptotically periodic Schrödinger equation with indefinite linear part. Commun. Contemp. Math.. 4, 763–776 (2002). Publisher Full Text

12. Li, GB, Wang, CH: The existence of nontrivial solutions to a semilinear elliptic system on R N without the Ambrosetti-Rabinowitz condition. Acta Math. Sci.. 30(6), 1917–1936 (2010)

13. Fernández Bonder, J, Pinasco, JP, Rossi, JD: Existence results for a Hamiltonian elliptic system with nonlinear boundary conditions. Electron. J. Differ. Equ.. 1999(40), 1–15 (electronic) (1999)

14. Fernández Bonder, J, Pinasco, JP, Rossi, JD: Infinitely many solutions for an elliptic system with nonlinear boundary conditions. Electron. J. Differ. Equ. Conf.. 6, 141–154 (2000)

15. Miyagaki, OH, Souto, MAS: Superlinear problems without Ambrosetti and Rabinowitz growth condition. J. Differ. Equ.. 245, 3628–3638 (2008). Publisher Full Text

16. Li, GB, Wang, CH: The existence of a nontrivial solution to a nonlinear elliptic problem of linking type without the Ambrosetti-Rabinowitz condition. Ann. Acad. Sci. Fenn., Ser. A 1 Math.. 36, 461–480 (2011)

17. Gilbarg, D, Trudinger, NS: Elliptic Partial Differential Equations of Second Order, Springer, Berlin (2001)

18. Taylor, ME: Partial Differential Equations: Basic Theory, Springer, New York (1996)

19. Thayer, J: Operadores auto-adjointos e equacões differenciais parciais, IMPA, Rio de Janeiro (1987)

20. Lions, JL, Magenes, E: Pronlémes aux limites non homogénes et applications, Dunod, Paris (1970)

21. Rabinowitz, P: Minimax methods in critical point theory with applications to differential equations. CBMS Regional Conference Series in Mathematics, Am. Math. Soc., Providence (1986)

22. Kryszewski, W, Szulkin, A: Generalized linking theorem with an application to semilinear Schrödinger equation. Adv. Differ. Equ.. 3, 441–472 (1998)