This article is concerned with the existence of solutions to the following nonlinear perturbed p-Laplacian system

{ − ε p Δ p u + V ( x ) | u | p − 2 u = K ( x ) | u | p * − 2 u + H u ( u , v ) ,   x ∈ ℝ N , − ε p Δ p v + V ( x ) | v | p − 2 v = K ( x ) | v | p * − 2 v + H v ( u , v ) ,   x ∈ ℝ N , u ( x ) ,   v ( x ) > 0 , u ( x ) ,   v ( x ) → 0   as | x | → ∞ ,

where Δ _p u = div(|∇u| ^p-2∇u) is the p-Laplacian operator, 1 < p < N and p* = Np/(N − p) is the critical exponent.

Throughout the article, we will assume that:

(V ₀) V ∈ C(ℝ ^N ), V (0) = inf V (x) = 0 and there exists b > 0 such that the set ν^b := {x ∈ ℝ ^N : V (x) < b} has finite Lebesgue measure;

(K ₀) K(x) ∈ C(ℝ ^N ), 0 < inf K ≤ sup K < ∞;

(H ₁) H ∈ C ¹(ℝ²) and H_s , H_t = o(|s| ^p-1 + |t| ^p-1) as |s| + |t| → 0;

(H ₂) there exist c > 0 and p < q < p* such that

| H s ( s , t ) | , | H t ( s , t ) | ≤ c ( 1 + | s | q - 1 + | t | q - 1 ) ;

(H ₃) There are a ₀ > 0, θ ∈ (p, p*) and α, β > p such that H(s, t) ≥ a ₀(|s|^α + |t|^β ) and 0 < θH(s, t) ≤ sH_s + tH_t .

Under the above mentioned conditions, we will get the following result.

Theorem 1. If (V ₀), (K ₀) and (H ₁)-(H ₃) hold, then for any σ > 0, there is ε_σ > 0 such that if ε < ε_σ , the problem (1.1) has at least one positive solution (u_ε , v_ε ) which satisfy

θ - p p θ ∫ ℝ N ( ε p | ∇ u ε | p + ε p | ∇ v ε | p + V ( x ) | u ε | p + V ( x ) | v ε | p ) ≤ σ ε N .

The scalar form of the problem (1.1) is as follows

- ε p Δ p u + V ( x ) | u | p - 2 u = K ( x ) | u | p * - 2 u + h ( x , u ) , x ∈ ℝ N .

The Equation (1.2) has been studied in many articles. The case p = 2 was investigated extensively under various hypotheses on the potential and the nonlinearity by many authors including Brézis and Nirenberg 1 , Ambrosetti 2 and Guedda and Veron 3 (see also their references) in bounded domains. As far as unbounded domains are concerned, we recall the work by Benci and Cerami 4 , Floer and Weistein 5 , Oh 6 , Clapp 7 , Del Pino and Felmer 8 , Cingolani and Lazzo 9 , Ding and Lin 10 . Especially, in 10 , the authors studied the Equation (1.2) in the case p = 2. In that article, they made the following assumptions:

(A ₁) V ∈ C(ℝ ^N ), min V = 0 and there is b > 0 such that the set ν^b := {x ∈ ℝ ^N : V (x) < b} has finite Lebesgue measure;

(A ₂) K(x) ∈ C(ℝ ^N ), 0 < inf K ≤ sup K < ∞

(B ₁) h ∈ C(ℝ ^N × ℝ) and h(x, u) = o(|u|) uniformly in x as |u| → 0;

(B ₂) there are c ₀ > 0, q < 2* such that |h(x, u)| ≤ c ₀(1 + |u| ^q-1) for all (x, u);

(B ₃) there are a ₀ > 0, p > 2 and µ > 2 such that H(x, u) = a ₀ |u|^p and µH(x, u) ≤ h(x, u)u for all (x, u), where H ( x , u ) = ∫ 0 u h ( x , s ) d s .

That article obtained the existence of at least one positive solution u_ε of least energy if the assumptions (A ₁)-(A ₂) and (B ₁)- (B ₃) hold.

For the Equation (1.2) in the case p ≠ 2, we recall some works. Garcia Azorero and Peral Alonso 11 considered (1.2) with ε ≤ 1, V (x) = µ, K(x) = 1, h(x, u) = 0 and proved that (1.2) has a solution if p ² ≤ N and µ ∈ (0, λ ₁), where λ ₁ is the first eigenvalue of the p-Laplacian. In 12 , Alves and Ding studied the same problem of 11 and obtained the multiplicity of positive solutions in bounded domain Ω ⊂ ℝ ^N . Moreover, Liu and Zheng 13 investigated (1.2) in ℝ ^N with ε = 1 and K(x) = 0. Under the sign-changing potential and subcritical p-superlinear nonlinearity, the authors got the existence result.

Motivated by some results found in 10 11 13 , a natural question arises whether existence of nontrivial solutions continues to hold for the p-Laplacian system with the critical nonlinearity in ℝ ^N .

The main difficulty in the case above mentioned is the lack of compactness of the energy functional associated to the system (1.1) because of unbounded domain ℝ ^N and critical nonlinearity. To overcome this difficulty, we make careful estimates and prove that there is a Palais-Smale sequence that has a strongly convergent sequence. The method or idea here is similar to the one of 10 . We can prove that the functional associated to (1.1) possesses (PS) _c condition at some energy level c. Furthermore, we prove the existence result by using the mountain pass theorem due to Rabinowitz 14 .

The main result in the present article concentrates on the existence of positive solutions to the system (1.1) and can be seen as a complement of the results developed in 10 11 13 .

This article is organized as follows. In Section 2, we give the necessary notations and preliminaries. Section 3 is devoted to the behavior of (PS) _c sequence and the mountain geometry structure. Finally, in Section 4, we prove the existence of nontrivial solution.

Let C 0 ∞ ( ℝ N ) denote the collection of smooth functions with compact support and D ^1,p (ℝ ^N ) be the completion of C 0 ∞ ( ℝ N ) under

| | u | | p = ∫ ℝ N | ∇ u | p d x .

We introduce the space

E ( ℝ N , V ) = { u ∈ W 1 , p ( ℝ N ) : ∫ ℝ N V ( x ) | u | p < ∞ }

equipped with the norm

| | u | | E = ∫ ℝ N ( | ∇ u | p + V ( x ) | u | p ) 1 p

and the space

E λ ( ℝ N , V ) = u ∈ W 1 , p ( ℝ N ) : ∫ ℝ N λ V ( x ) | u | p < ∞ , λ > 0

under

| | u | | λ = ( ∫ ℝ N | ∇ u | p + λ V ( x ) | u | p ) ) 1 p .

Observe that ‖ · ‖ _E is equivalent to the one ‖ · ‖ _λ for each λ > 0. It follows from (V ₀) that E(ℝ ^N , V) continuously embeds in W ^1,p (ℝ ^N ).

Set B = E_λ × E_λ and | | ( u , v ) | | λ = | | u | | λ p + | | v | | λ p for any (u, v) ∈ B. Let λ = ε ^-p in the system (1.1), then (1.1) is changed into

{ − Δ p u + λ V ( x ) | u | p − 2 u = λ K ( x ) | u | p * − 2 u + λ H u ( u , v ) ,   ∈ ℝ N , − Δ p v + λ V ( x ) | v | p − 2 v = λ K ( x ) | v | p * − 2 v + λ H v ( u , v ) ,   x ∈ ℝ N , u ( x ) ,   v ( x ) > 0 , u ( x ) ,   v ( x ) → 0 ,   as   | x | → ∞ .

In order to prove Theorem 1, we only need to prove the following result.

Theorem 2. Let (V ₀), (K ₀) and (H ₁)-(H ₃) be satisfied. Then for any σ > 0, there exists Λ _σ > 0 such that if λ ≥ Λ _σ , the system (2.1) has at least one least energy solution (u_λ , v_λ ) satisfying

θ - p p θ ∫ ℝ N ( | ∇ u λ | p + | ∇ v λ | p + λ V ( x ) ( | u λ | p + | v λ | p ) ) ≤ σ λ 1 - N p .

The energy functional associated with (2.1) is defined by

I λ ( u , v ) = 1 p ∫ ℝ N ( | ∇ u | p + λ V ( x ) | u | p + | ∇ v | p + λ V ( x ) | v | p ) - λ p * ∫ ℝ N K ( x ) ( | u | p * + | v | p * ) - λ ∫ ℝ N H ( u , v ) = 1 p | | ( u , v ) | | λ p - λ ∫ ℝ N G ( u , v ) ,

where G ( u , v ) = 1 p * K ( x ) ( | u | p * + | v | p * ) + H ( u , v ) .

From the assumptions of Theorem 2, standard arguments 14 show that I_λ ∈ C ¹(B, ℝ) and its critical points are the weak solutions of (2.1).

In this section, we will recall and prove some lemmas which are crucial in the proof of the main result.

Lemma 3.1. Let the assumptions of Theorem 2 be satisfied. If the sequence {(u_n , v_n )} ⊂ B is a (PS) _c sequence for I_λ , then we get that c ≥ 0 and {(u_n , v_n )} is bounded in the space B.

Proof. One has

I λ ( u n , v n ) - 1 θ I λ ' ( u n , v n ) ( u n , v n ) = 1 p | | ( u n , v n ) | | λ p - λ p * ∫ ℝ N K ( x ) ( | u n | p * + | v n | p * ) - λ ∫ ℝ N H ( u n , v n ) - 1 θ | | ( u n , v n ) | | λ p - λ ∫ ℝ N K ( x ) ( | u n | p * + | v n | p * ) - λ ∫ ℝ N ( u n H s ( u n , v n ) + v n H t ( u n , v n ) ) = 1 p - 1 θ | | ( u n , v n ) | | λ p + 1 θ - 1 p * λ ∫ ℝ N K ( x ) ( | u n | p * + | v n | p * ) + λ ∫ ℝ N 1 θ ( u n H s ( u n , v n ) + v n H t ( u n , v n ) ) - H ( u n , v n )

By the assumptions (K ₀) and (H ₃), we have

I λ ( u n , v n ) - 1 θ I λ ′ ( u n , v n ) ( u n , v n ) ≥ 1 p - 1 θ | | ( u n , v n ) | | λ p .

Together with I_λ (u_n , v_n ) → c and I λ ′ ( u n , v n ) → 0 as n → ∞, we easily obtain that the (PS) _c sequence is bounded in B and the energy level c ≥ 0. □

From Lemma 3.1, there exists (u, v) ∈ B such that (u_n , v_n ) ⇀ (u, v) in B. Furthermore, passing to a subsequence, we have u_n → u and v_n → v in L l o c d ( ℝ N ) for any d ∈ [p, p*) and u_n → u, v_n → v a.e. in ℝ ^N .

Lemma 3.2. Let d ∈ [p, p*). There exists a subsequence { ( u n j , v n j ) } such that for any ε > 0, there is r_ε > 0 with

lim i → ∞ sup ∫ B i \ B r ( | u n i | d + | v n i | d ) ≤ ε

for any r ≥ r_ε , where B_r := {x ∈ ℝ ^N : |x| ≤ r}.

Proof. The proof of Lemma 3.2 is similar to the one of Lemma 3.2 of 10 , so we omit it. □

Let η ∈ C ^∞(ℝ⁺) be a smooth function satisfying 0 ≤ η(t) ≤ 1, η(t) = 1 if t ≤ 1 and η(t) = 0 if t ≥ 2. Define ũ j ( x ) = η ( 2 | x | / j ) u ( x ) , ṽ j ( x ) = η ( 2 | x | / j ) v ( x ) . It is obvious that

| | u - ũ j | | λ → 0 and | | v - ṽ j | | λ → 0 as  j → ∞ .

Lemma 3.3. One has

lim j → ∞ ∫ ℝ N ( H s ( u n j , v n j ) - H s ( u n j - ũ j , v n j - ṽ j ) - H s ( ũ j , ṽ j ) ) φ = 0

and

lim j → ∞ ∫ ℝ N ( H t ( u n j , v n j ) - H t ( u n j - ũ j , v n j - v j ) - H t ( ũ j , ṽ j ) ) ψ = 0

uniformly in (φ, ψ) ∈ B with ‖(φ, ψ‖ _B ≤ 1.

Proof. From the assumptions (H ₁)-(H ₂) and Lemma 3.2, we have

  lim j → ∞ sup ∫ ℝ N ( H s ( u n j ,   v n j ) − H s ( u n j − u ˜ j ,   v n j − v ˜ j ) − H s ( u ˜ j ,   v ˜ j ) ) φ = lim j → ∞ sup ∫ B j ( H s ( u n j ,   v n j ) − H s ( u n j − u ˜ j ,   v n j − v ˜ j ) − H s ( u ˜ j ,   v ˜ j ) ) φ = lim j → ∞ sup ∫ B j \ B r ( H s ( u n j ,   v n j ) − H s ( u n j − u ˜ j ,   v n j − v ˜ j ) − H s ( u ˜ j ,   v ˜ j ) ) φ ≤ c lim j → ∞ sup ∫ B j \ B r ( | u n j | p − 1 + | v n j | p − 1 + | u n j | q − 1 + | v n j | q − 1 + | u ˜ j | p − 1 + | v ˜ j | p − 1   + | u ˜ j | q − 1 + | v ˜ j | q − 1 + | u n j − u ˜ j | p − 1 + | v n j − v ˜ j | p − 1 + | u n j − u ˜ j | q − 1 + | v n j − v ˜ j | q − 1 ) φ ≤ c 1 lim j → ∞ sup ∫ B j \ B r ( | u n j | p − 1 + | v n j | p − 1 + | u ˜ j | p − 1 + | v ˜ j | p − 1 ) φ   + c 2 lim j → ∞ sup ∫ B j \ B r ( | u n j | q − 1 + | v n j | q − 1 + | u ˜ j | q − 1 + | v ˜ j | q − 1 ) φ

By Hölder inequality and Lemma 3.2, it follows that

lim j → ∞ sup ∫ B j \ B r | u n j | p - 1 | φ | ≤ lim j → ∞ sup ∫ B j \ B r | u n j | p p - 1 p ∫ B j \ B r | φ | p 1 p ≤ lim j → ∞ sup ∫ B j \ B r | u n j | p p - 1 p ∫ ℝ N | φ | p 1 p ≤ lim j → ∞ sup ∫ B j \ B r | u n j | p p - 1 p = 0

and

lim j → ∞ sup ∫ B j \ B r | u n j | p - 1 | φ | ≤ lim j → ∞ sup ∫ B j \ B r | u n j | p q - 1 p ∫ B j \ B r | φ | q 1 q ≤ lim j → ∞ sup ∫ B j \ B r | u n j | q q - 1 q ∫ ℝ N | φ | q 1 q ≤ lim j → ∞ sup ∫ B j \ B r | u n j | q q - 1 q = 0

Similarly, we get

lim j → ∞ sup ∫ B j \ B r ( | v n j | p - 1 | + | ũ j | p - 1 + | ṽ j | p - 1 ) φ = 0

and

lim j → ∞ sup ∫ B j \ B r ( | v n j | q - 1 | + | ũ j | q - 1 + | ṽ j | q - 1 ) φ = 0 .

Thus

lim j → ∞ ∫ ℝ N ( H s ( u n j , v n j ) - H s ( u n j - ũ j , v n j - ṽ j ) - H s ( ũ j , ṽ j ) ) φ = 0 .

From the similar argument, we also get

lim j → ∞ ∫ ℝ N ( H t ( u n j , v n j ) - H t ( u n j - ũ j , v n j - ṽ j ) - H t ( ũ j , ṽ ′ j ) ) ψ = 0 .

□

Lemma 3.4. One has along a subsequence

I λ ( u n - ũ n , v n - ṽ n ) → c - I λ ( u , v )

and

I λ ′ ( u n - ũ n , v n - ṽ n ) → 0 in  B - 1 ( the dual space of  B ) .

Proof. From the Lemma 2.1 of 15 and the argument of 16 , we have

I λ ( u n - ũ n , v n - ṽ n ) = 1 p ∫ ℝ N ( | ∇ u n - ∇ ũ n | p + λ V ( x ) | u n - ũ n | p + | ∇ v n - ∇ ṽ n | p + λ V ( x ) | v n - ṽ n | p ) - λ p * ∫ ℝ N K ( x ) ( | u n - ũ n | p * + | v n - ṽ n | p * ) - λ ∫ ℝ N H ( u n - ũ n , v n - ṽ n ) = I λ ( u n , v n ) - I λ ( ũ n , ṽ n ) + λ p * ∫ ℝ N K ( x ) ( ( | u n | p * - | u n - ũ n | p * - | ũ n | p * ) + ( | v n | p * - | v n - ṽ n | p * - | ṽ n | p * ) ) + λ ∫ ℝ N ( H ( u n , v n ) - H ( u n - ũ n , v n - ṽ n ) - H ( ũ n , ṽ n ) ) + o ( 1 ) .

By (3.1) and the similar idea of proving the Brézis-Lieb Lemma 17 , it is easy to get

lim n → ∞ ∫ ℝ N K ( x ) ( ( | u n | p * - | u n - ũ n | p * - | ũ n | p * ) + ( | v n | p * - | v n - ṽ n | p * - | ṽ n | p * ) ) = 0

and

lim n → ∞ ∫ ℝ N ( H ( u n , v n ) - H ( u n - ũ n , v n ṽ n ) - H ( ũ n , ṽ n ) ) = 0 .

In connection with the fact I_λ (u_n , v_n ) → c and I λ ( ũ n , ṽ n ) → I λ ( u , v ) , we obtain

I λ ( u n - ũ n , v n - ṽ n ) → c - I λ ( u , v ) .

In the following, we will verify the fact I λ ′ ( u n - ũ n , v n - ṽ n ) → 0 .

For any (φ, ψ) ∈ B, it follows that

  I λ ' ( u n − u ˜ n , v n − v ˜ n ) ( φ , ψ ) = I λ ' ( u n , v n ) ( φ , ψ ) − I λ ' ( u ˜ n , v ˜ n ) ( φ , ψ )   + λ ∫ ℝ N K ( x ) [ ( | u n | p * − 2 u n − | u n − u ˜ n | p * − 2 ( u n − u ˜ n ) − | u ˜ n | p * − 2 u ˜ n ) φ   + ( | v n | p * − 2 v n − | v n − v ˜ n | p * − 2 ( v n − v ˜ n ) − | v ˜ n | p * − 2 v ˜ n ) ψ ]   + λ ∫ ℝ N [ ( H s ( u n ,   v n ) − H s ( u n − u ˜ n ,   v n − v ˜ n ) − H s ( u ˜ n ,   v ˜ n ) ) φ   + ( H t ( u n ,   v n ) − H t ( u n − u ˜ n ,   v n − v ˜ n ) − H t ( u ˜ n ,   v ˜ n ) ) ψ ] + o ( 1 ) .

Standard argument shows that

lim n → ∞ ∫ ℝ N K ( x ) ( | u n | p * - 2 u n - | u n - ũ n | p * - 2 ( u n - ũ n ) - | ũ n | p * - 2 ũ n ) φ = 0

and

lim n → ∞ ∫ ℝ N K ( x ) ( | v n | p * - 2 v n - | v n - ṽ n | p * - 2 ( v n - ṽ n ) - | ṽ n | p * - 2 ṽ n ) ψ = 0

uniformly in ‖φ, ψ)‖ _B ≤ 1.

By Lemma 3.3, we have

lim n → ∞ ∫ ℝ N ( H s ( u n , v n ) - H s ( u n - ũ n , v n - ṽ n ) - H s ( ũ n , ṽ n ) ) φ = 0

and

lim n → ∞ ∫ ℝ N ( H t ( u n , v n ) - H t ( u n - ũ n , v n - ṽ n ) - H t ( ũ n , ṽ n ) ) ψ = 0

uniformly in ‖(φ, ψ)‖ _B ≤ 1. From the facts above mentioned, we obtain

I λ ′ ( u n - ũ n , v n - ṽ n ) → 0 in  B - 1 .

□

Let u n 1 = u n - ũ n , v n 1 = v n - ṽ n , then u n - u = u n 1 + ( ũ n - u ) , v n - v = v n 1 + ( ṽ n - v ) . From (3.1), we get (u_n , v_n ) → (u, v) in B if and only if ( u n 1 , v n 1 ) → ( 0 , 0 ) in B.

Observe that

I λ ( u n 1 , v n 1 ) - 1 p I λ ' ( u n 1 , v n 1 ) ( u n 1 , v n 1 ) = 1 p - 1 p * λ ∫ ℝ N K ( x ) ( | u n 1 | p * + | v n 1 | p * ) + λ ∫ ℝ N 1 p ( u n 1 H s ( u n 1 , v n 1 ) + v n 1 H t ( u n 1 , v n 1 ) ) - H ( u n 1 , v n 1 ) ≥ λ N ∫ ℝ N K ( x ) ( | u n 1 | p * + | v n 1 | p * ) ≥ λ N K min ∫ ℝ N ( | u n 1 | p * + | v n 1 | p * ) ,

where K min = inf x ∈ ℝ N K ( x ) > 0 .

Thus by Lemma 3.4, we get

| | ( u n 1 , v n 1 ) | | p * p * ≤ N ( c - I λ ( u , v ) ) λ K min + o ( 1 ) .

Now, we consider the energy level of the functional I_λ below which the (PS) _c condition hold.

Let V_b (x):= max{V (x), b}, where b is the positive constant in the assumption (V ₀). Since the set ν_b has finite measure and u n 1 , v n 1 → 0 in L loc p ( ℝ N ) , we get

∫ ℝ N V ( x ) ( | u n 1 | p + | v n 1 | p ) = ∫ ℝ N V b ( x ) ( | u n 1 | p + | v n 1 | p ) + o ( 1 ) .

From (K ₀), (H ₁)-(H ₃) and Young inequality, there is C_b > 0 such that

∫ ℝ N ( K ( x ) ( | u | p * + | v | p * ) + u H s ( u , v ) + v H t ( u , v ) ) ≤ b ( | | u | | p p + | | v | | p p ) + C b ( | | u | | p * p * + | | v | | p * p * ) .

Let S be the best Sobolev constant of the immersion

S | | u | | p * p ≤ ∫ ℝ N | ∇ u | p for all  u ∈ W 1 , p ( ℝ N ) .

Lemma 3.5. Let the assumptions of Theorem 2 be satisfied. There exists α ₀ > 0 independent of λ such that, for any (PS) _c sequence {(u_n , v_n )} ⊂ B for I_λ with (u_n , v_n ) ⇀ (u, v), either (u_n , v_n ) → (u, v) or c - I λ ( u , v ) ≥ α 0 λ 1 - N p .

Proof. Assume that (u_n , v_n ) ↛ (u, v), then

lim inf n → ∞ | | ( u n 1 ,   v n 1 ) | | λ > 0

and

c - J λ ( u , v ) > 0 .

By the Sobolev inequality, (3.4) and (3.5), we get

S ( | | u n 1 | | p * p + | | v n 1 | | p * p ) ≤ ∫ ℝ N ( | ∇ u n 1 | p + | ∇ v n 1 | p ) = ∫ ℝ N ( | ∇ u n 1 | p + λ V ( x ) | u n 1 | p + | ∇ v n 1 | p + λ V ( x ) | v n 1 | p ) - λ ∫ ℝ N V ( x ) ( | u n 1 | p + | v n 1 | p ) = λ ∫ ℝ N K ( x ) ( | u n 1 | p * + | v n 1 | p * ) + u n 1 H s ( u n 1 , v n 1 ) + v n 1 H t ( u n 1 , v n 1 ) - λ ∫ ℝ N V ( x ) ( | u n 1 | p + | v n 1 | p ) + o ( 1 ) ≤ λ b ( | | u n 1 | | p p + | | v n 1 | | p p ) + λ C b ( | | u n 1 | | p * p * + | | v n 1 | | p * p * ) - λ b ( | | u n 1 | | p p + | | v n 1 | | p p ) + o ( 1 ) = λ C b ( | | u n 1 | | p * p * + | | v n 1 | | p * p * ) + o ( 1 ) .