Shipping Option Name Missing from IPN - paypal

Question

What would cause the shipping option chosen by the customer (along with some related fields) to not be returned in the IPN?
We are using an encrypted 'cart upload' command combined with the Instant Update API to provide dynamic shipping options based on the customer's address (explained here ). In some (not all) cases, the shipping_option_name and some related values are not being returned in the IPN. Additionally, the name of the shipping option selected by the user is not available from the PayPal interface when we view transaction details.
Note that this happens in only some cases, and we cannot find a common cause among the failures.
UPDATE 1/29/14 - We have noticed that the shipping option name is being recorded for ALL transactions in which the Counterparty Status is "Unregistered," and about 25% of the transactions for which the Counterparty Status is something else (e.g., Confirmed, Non-Confirmed, Verified, etc.).
We have set up logging to track outgoing Instant Update data and incoming IPN data. In cases where the shipping_option_name is missing from the IPN, the following fields are also missing:
Shipping_calculation_mode
shipping_option_name
insurance_option_selected
shipping_option_amount
insurance_amount
receipt_id
shipping_is_default
I'm pasting our Instant Update response and the PayPal IPN for two very similar transactions below. Both transactions purchase the same product, have the same shipping price, and the customers live in the same state. The first transaction successfully returned the shipping option name in the IPN and the second transaction did not. Sensitive values have been replaced with XX. Note the Instant Update response is identical between the two transactions, yet some values are missing from the IPN for the second transaction.
First Transaction - Successfully records shipping option name
------------
OUR INSTANT UPDATE RESPONSE
-----------
METHOD=CallbackResponse
&L_SHIPPINGOPTIONLABEL0=Ground
&L_SHIPPINGOPTIONAMOUNT0=16.95
&L_TAXAMT0=0
&L_SHIPPINGOPTIONISDEFAULT0=true
&L_SHIPPINGOPTIONLABEL1=3-Day
&L_SHIPPINGOPTIONAMOUNT1=45
&L_TAXAMT1=0
&L_SHIPPINGOPTIONISDEFAULT1=false
&L_SHIPPINGOPTIONLABEL2=2-Day
&L_SHIPPINGOPTIONAMOUNT2=60
&L_TAXAMT2=0
&L_SHIPPINGOPTIONISDEFAULT2=false
&L_SHIPPINGOPTIONLABEL3=Next Day - Standard (Afternoon)
&L_SHIPPINGOPTIONAMOUNT3=85
&L_TAXAMT3=0
&L_SHIPPINGOPTIONISDEFAULT3=false
&L_SHIPPINGOPTIONLABEL4=Next Day - Priority (Morning)
&L_SHIPPINGOPTIONAMOUNT4=100
&L_TAXAMT4=0
&L_SHIPPINGOPTIONISDEFAULT4=false
------------
IPN DATA FROM PAYPAL
-----------
[p] => store/paypal-checkout-response
[Shipping_calculation_mode] => Callback
[mc_gross] => 132.15
[protection_eligibility] => Eligible
[address_status] => confirmed
[item_number1] => 18827
[payer_id] => XX
[tax] => 0.00
[address_street] => XX
[payment_date] => 09:00:47 Jan 23, 2014 PST
[payment_status] => Completed
[charset] => windows-1252
[shipping_option_name] => Ground
[address_zip] => XX
[mc_shipping] => 16.95
[mc_handling] => 0.00
[first_name] => XX
[mc_fee] => 4.13
[address_country_code] => US
[address_name] => XX
[notify_version] => 3.7
[insurance_option_selected] => 0
[custom] => XX
[payer_status] => unverified
[business] => XX
[address_country] => United States
[shipping_option_amount] => 16.95
[num_cart_items] => 1
[mc_handling1] => 0.00
[address_city] => XX
[verify_sign] => XX
[payer_email] => XX
[mc_shipping1] => 0.00
[tax1] => 0.00
[txn_id] => XX
[payment_type] => instant
[last_name] => XX
[address_state] => MO
[item_name1] => End Caps
[receiver_email] => XX
[payment_fee] => 4.13
[quantity1] => 1
[insurance_amount] => 0.00
[receiver_id] => XX
[txn_type] => cart
[mc_gross_1] => 115.20
[mc_currency] => USD
[residence_country] => US
[receipt_id] => XX
[transaction_subject] => XX
[payment_gross] => 132.15
[shipping_is_default] => 1
[ipn_track_id] => XX
Second transaction - Shipping option name was not recorded or returned by IPN
------------
OUR INSTANT UPDATE RESPONSE
-----------
METHOD=CallbackResponse
&L_SHIPPINGOPTIONLABEL0=Ground
&L_SHIPPINGOPTIONAMOUNT0=16.95
&L_TAXAMT0=0
&L_SHIPPINGOPTIONISDEFAULT0=true
&L_SHIPPINGOPTIONLABEL1=3-Day
&L_SHIPPINGOPTIONAMOUNT1=45
&L_TAXAMT1=0
&L_SHIPPINGOPTIONISDEFAULT1=false
&L_SHIPPINGOPTIONLABEL2=2-Day
&L_SHIPPINGOPTIONAMOUNT2=60
&L_TAXAMT2=0
&L_SHIPPINGOPTIONISDEFAULT2=false
&L_SHIPPINGOPTIONLABEL3=Next Day - Standard (Afternoon)
&L_SHIPPINGOPTIONAMOUNT3=85
&L_TAXAMT3=0
&L_SHIPPINGOPTIONISDEFAULT3=false
&L_SHIPPINGOPTIONLABEL4=Next Day - Priority (Morning)
&L_SHIPPINGOPTIONAMOUNT4=100
&L_TAXAMT4=0
&L_SHIPPINGOPTIONISDEFAULT4=false
------------
IPN DATA FROM PAYPAL
-----------
[p] => store/paypal-checkout-response
[mc_gross] => 132.15
[protection_eligibility] => Eligible
[address_status] => confirmed
[item_number1] => 18827
[payer_id] => XX
[tax] => 0.00
[address_street] => XX
[payment_date] => 07:11:37 Jan 23, 2014 PST
[payment_status] => Completed
[charset] => windows-1252
[address_zip] => XX
[mc_shipping] => 16.95
[mc_handling] => 0.00
[first_name] => XX
[mc_fee] => 4.13
[address_country_code] => US
[address_name] => XX
[notify_version] => 3.7
[custom] => XX
[payer_status] => verified
[business] => XX
[address_country] => United States
[num_cart_items] => 1
[mc_handling1] => 0.00
[address_city] => XX
[verify_sign] => XX
[payer_email] => XX
[mc_shipping1] => 0.00
[tax1] => 0.00
[txn_id] => XX
[payment_type] => instant
[last_name] => XX
[address_state] => MO
[item_name1] => End Caps
[receiver_email] => XX
[payment_fee] => 4.13
[quantity1] => 1
[receiver_id] => XX
[txn_type] => cart
[mc_gross_1] => 115.20
[mc_currency] => USD
[residence_country] => US
[transaction_subject] => XX
[payment_gross] => 132.15
[ipn_track_id] => XX
Any help would be greatly appreciated!

Answer 1

It all comes down to the DoExpressCheckoutPayment requestion. You need to make sure to pass all the details about the shipping and address in this final call or it won't get included in the transaction details which means it won't be available in IPN.