Trouble installing TIA portal S7 V17 - plc

I'm trying to install Siemens TIA portal V17 but get a setup error.
2022.11.03 17:16:20 - Year.Month.Day - Log start
BUNDLE: SIMATIC TIA Portal V17 WinCC Prof Ed
SEBU Version: Professional:TIA.rgs.RTM_17000000:V17.00.00.00_43.02.00.01
CHANGES: (+)DotNet_Updater;(o)SQLNCLI;(o)INSTSQL2017EXP_SCADA;(+)SYMLINK;(+)SYMLINKADD;(+)OPCCORECOMP64;(+)NCMGPRS64;(+)NCMFWL64;(+)WinCC ACE;(+)WinCC Common Archiving;(+)OPC;(+)Runtime Interfaces;(+)HMIProvider;(+)SECON;(+)S7DOS64;(+)S7DOS;(+)S7EDB;(+)TIA_Portal;(+)MU_Client;(+)HWCN-HWCOL;(+)SUPBASEH02;(+)SUPBASEH03;(+)SUPBASEH04;(+)SUPBASETO01;(+)SUPBASETO02;(+)SUPBASEWCF01;(+)TIAP_SIMATIC;(+)TIAP_SIMATIC32;(+)STEP7;(+)WinCC Runtime Advanced Simulator;(+)PTSymLib;(+)PTTouchInput;(+)HM_AllEditions;(+)HM_NoBasic;(+)STOBS;(o)UMC64;(+)UMTRAYICON;(o)LicenseManager;(+)STOP_ALMSERVICE;(o)AlmPanelPlugin64;(o)Prosave;(+)TIA_ES;(+)TIA_ES32;(+)WinCC_Tr_Mandatory;(+)SCADA-RT;(+)RT_Pro_Sim;(+)CoRtHmiRTmDriver64;(+)HMIRTMV11;(+)RT_Adv_Sim;(+)TIACOMPCHECK;(+)S7FPLUS;(+)TIA_Opns;(+)OptionVCI;(+)ETWEventCollector;(+)TelemetryConnector;(+)OPCUA_Client;(o)TIAADMIN;(o)AWB_SOFT;(o)AWB_ALM;(o)AWB_TIAUMAC;(+)MUSERVER
There are NO severe messages before the installation
17:16:20 | BEGIN. Execution started.
17:16:26 | OK RunningProcesses () Complete
17:16:26 | OK InventoryBefore (A) Complete
17:16:26 | OK Citamis (Before) Complete
17:16:44 | OK SecurityControlFeature (ReturnFeatures) Complete
17:16:44 | OK StartApplBefore (TIAP17::[INSTALLDIR]BIN\SIEMENS.AUTOMATION.OBJECTFRAME.FILESTORAGE.SERVER.EXE)Complete
17:16:44 | OK StartApplBefore (TIAADMIN::[INSTALLDIR]SERVER\NODE.EXE)Complete
17:16:44 | OK StartApplBefore (TIAP17::[INSTALLDIR]X64\BIN\SIEMENS.AUTOMATION.OBJECTFRAME.FILESTORAGE.SERVER.EXE)Complete
17:16:44 | OK ClassicCompPrepare () Complete
17:16:44 | OK ProductRegistration (Before) Complete
17:16:44 | OK ClassicCompAfterUninstall () Complete
17:16:45 | OK SetupUnit (DotNet_Updater) Complete Foreign AddLocal
17:16:45 | OK Checkpoint (RE) Complete
17:16:45 | OK SetupUnit (SQLNCLI) Complete Foreign Reinstall
17:16:48 |ERROR AdsWorkerClassicProduct::OnWorkerCompleted(): - ClassicProduct setup failed due to unknown error
17:16:48 | FAIL! SetupUnit (INSTSQL2017EXP_SCADA) Failed ClassicProductReinstall
17:16:48 | FAIL! Checkpoint (E) Complete
17:16:48 |ERROR ExecutionEngine::StartNextStepAsync(): - Execution has been cancelled. Probably because of some process(es) running on the machine
17:16:48 | SKIP! SetupUnit (SYMLINK) Skip Foreign AddLocal
17:16:48 | SKIP! SetupUnit (SYMLINKADD) Skip Foreign AddLocal
17:16:48 | SKIP! SetupUnit (OPCCORECOMP64) Skip Foreign AddLocal
17:16:48 | SKIP! SetupUnit (NCMGPRS64) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (NCMFWL64) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (WinCC ACE) Skip
17:16:48 | SKIP! SetupUnit (WinCC ACE) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (WinCC ACE) Skip
17:16:48 | SKIP! SetupUnit (WinCC Common Archiving) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (OPC) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (Runtime Interfaces) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (HMIProvider) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (SECON) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (S7DOS64) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (S7DOS) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (S7EDB) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (TIA_Portal) Skip
17:16:48 | SKIP! SetupUnit (TIA_Portal) Skip PlusMain AddLocal
17:16:48 | SKIP! SecurityControl (TIA_Portal) Skip
17:16:48 | SKIP! SetupUnit (MU_Client) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (HWCN-HWCOL) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASEH02) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASEH03) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASEH04) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASETO01) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASETO02) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (SUPBASEWCF01) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (TIAP_SIMATIC) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (TIAP_SIMATIC32) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (STEP7) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (WinCC Runtime Advanced Simulator)Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (PTSymLib) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (PTTouchInput) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (HM_AllEditions) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (HM_NoBasic) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (STOBS) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (UMC64) Skip
17:16:48 | SKIP! SetupUnit (UMC64) Skip PlusMain Reinstall
17:16:48 | SKIP! SecurityControl (UMC64) Skip
17:16:48 | SKIP! SetupUnit (UMTRAYICON) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SecurityControl (LicenseManager) Skip
17:16:48 | SKIP! SetupUnit (LicenseManager) Skip ClassicProductReinstall
17:16:48 | SKIP! SecurityControl (LicenseManager) Skip
17:16:48 | SKIP! SetupUnit (STOP_ALMSERVICE) Skip Foreign AddLocal
17:16:48 | SKIP! SetupUnit (AlmPanelPlugin64) Skip ClassicComponentReinstall
17:16:48 | SKIP! SetupUnit (Prosave) Skip ClassicProductReinstall
17:16:48 | SKIP! SetupUnit (TIA_ES) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (TIA_ES32) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (WinCC_Tr_Mandatory) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SecurityControl (SCADA-RT) Skip
17:16:48 | SKIP! SetupUnit (SCADA-RT) Skip PlusMain AddLocal
17:16:48 | SKIP! SecurityControl (SCADA-RT) Skip
17:16:48 | SKIP! SetupUnit (RT_Pro_Sim) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (CoRtHmiRTmDriver64) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (HMIRTMV11) Skip
17:16:48 | SKIP! SetupUnit (HMIRTMV11) Skip PlusMain AddLocal
17:16:48 | SKIP! SecurityControl (HMIRTMV11) Skip
17:16:48 | SKIP! SetupUnit (RT_Adv_Sim) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (TIACOMPCHECK) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (S7FPLUS) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SecurityControl (TIA_Opns) Skip
17:16:48 | SKIP! SetupUnit (TIA_Opns) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SecurityControl (TIA_Opns) Skip
17:16:48 | SKIP! SetupUnit (OptionVCI) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SetupUnit (ETWEventCollector) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SetupUnit (TelemetryConnector) Skip ClassicComponentAddLocal
17:16:48 | SKIP! SecurityControl (OPCUA_Client) Skip
17:16:48 | SKIP! SetupUnit (OPCUA_Client) Skip PlusOptionalAddLocal
17:16:48 | SKIP! SecurityControl (OPCUA_Client) Skip
17:16:48 | SKIP! SetupUnit (TIAADMIN) Skip PlusMain Reinstall
17:16:48 | SKIP! SetupUnit (AWB_SOFT) Skip PlusOptionalReinstall
17:16:48 | SKIP! SetupUnit (AWB_ALM) Skip PlusOptionalReinstall
17:16:48 | SKIP! SetupUnit (AWB_TIAUMAC) Skip PlusOptionalReinstall
17:16:48 | SKIP! SetupUnit (MUSERVER) Skip PlusMain AddLocal
17:16:48 | SKIP! ClassicCompAfterInstall () Skip
17:16:48 | SKIP! ClassicCompRollback () Skip
17:16:48 | SKIP! HspInstall () Skip
17:16:48 | SKIP! MspInstall () Skip
17:16:48 | SKIP! StartApplAfter (TIAP17::[INSTALLDIR]BIN\INSTALLROOTCERTIFICATE.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAADMIN::[INSTALLDIR]SERVER\NODE.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAP17::[INSTALLDIR]BIN\SIEMENS.AUTOMATION.GENERATOR.FRAMEWORK.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAADMIN::TIAADMINNOTIFIER.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAP17::C:\WINDOWS\SYSWOW64\S7EPATSX.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAP17::C:\WINDOWS\SYSWOW64\S7EPAESX.EXE)Skip
17:16:48 | SKIP! StartApplAfter (TIAP17::[INSTALLDIR]BIN\SIEMENS.SIMATIC.HWCN.INTERPRETER.IG.GSD.GENERATOR.EXE)Skip
17:16:48 | SKIP! ProductRegistration (After) Skip
17:16:48 | SKIP! InventoryAfter (Z) Skip
17:16:48 | SKIP! LicenseTransfer (False) Skip
17:16:48 | SKIP! SetupReport () Skip
17:16:49 | OK SetupIsComplete () Complete
17:16:49 | OK Citamis (After) Complete
17:16:49 | OK SetupSummary () Complete
17:16:49 | END. Execution completed.
Intermediate ExitCode after execution queue : 1603 - Exit code means: Installation failed (InstallFailure)
Errors: 2
Warnings: 0
Exit installation with global errorcode : 1603 - Exit code means: Installation failed (InstallFailure)


Some possible ways to solve:
Try to deactivate your anti virus program during installation
Maybe you haven't enough rights to install system components
Check the system requirements on https://support.industry.siemens.com/cs/document/109795525/delivery-release-simatic-wincc-v17?dti=0&lc=en-US

Related

How can I read from port IO in Simics?

I can see from help-search that there's a "<port_space>.read", but I don't know how to find the <port_space> name for the CPU.

port_space attribute of a processor core points to a memory space object that is used for port accesses. You can access to memory space object directly using get/set (for non-architectural access), read/write (for architectural access) for example:
simics> board.mb.cpu0.core[0][0]->port_space
"board.mb.cpu0.ports_proxy[0][0]"
simics> board.mb.cpu0.ports_proxy[0][0].write 0xcf9 0xff size=1
[board.mb.nb.pci_bus info] sending hot_reset
You can also traverse the memory space to find what is exactly mapped there:
simics> board.mb.cpu0.ports_proxy[0][0].map
+---------+-------------------+--+------+------+------+----+-----+----+
| Base|Object |Fn|Offset|Length|Target|Prio|Align|Swap|
+---------+-------------------+--+------+------+------+----+-----+----+
|-default-|board.mb.port_mem_m| | 0x0| | | | | |
+---------+-------------------+--+------+------+------+----+-----+----+
simics> board.mb.port_mem_m.map
+----+-----------------+--+------+-------+------+----+-----+----+
|Base|Object |Fn|Offset| Length|Target|Prio|Align|Swap|
+----+-----------------+--+------+-------+------+----+-----+----+
| 0x0|board.mb.port_mem| | 0x0|0x10000| | 0| | |
+----+-----------------+--+------+-------+------+----+-----+----+
simics> board.mb.port_mem.map
+---------+-------------------------------+--+------+------+------+----+-----+----+
| Base|Object |Fn|Offset|Length|Target|Prio|Align|Swap|
+---------+-------------------------------+--+------+------+------+----+-----+----+
| 0x402|board.mb.conf | | 0x0| 0x1| | 0| | |
| 0x510|board.mb.conf | 3| 0x0| 0x2| | 0| | |
| 0x511|board.mb.conf | 4| 0x0| 0x1| | 0| | |
| 0xcf8|board.mb.nb.bridge.bank.io_regs| | 0xcf8| 0x4| | 0| | |
| 0xcf9|board.mb.sb.cf9 | | 0x0| 0x1| | 0| | |
| 0xcfc|board.mb.nb.bridge.bank.io_regs| | 0xcfc| 0x4| | 0| | |
| 0xcfd|board.mb.nb.bridge.bank.io_regs| | 0xcfd| 0x2| | 0| | |
| 0xcfe|board.mb.nb.bridge.bank.io_regs| | 0xcfe| 0x2| | 0| | |
| 0xcff|board.mb.nb.bridge.bank.io_regs| | 0xcff| 0x1| | 0| | |
| 0xfff0|board.mb.conf | | 0x0| 0x1| | 0| | |
| 0xfff1|board.mb.conf | 1| 0x0| 0x1| | 0| | |
| 0xfff2|board.mb.conf | 2| 0x0| 0x2| | 0| | |
| 0xfff4|board.mb.shadow | | 0x0| 0x1| | 0| | |
| 0xfff5|board.mb.shadow | | 0x1| 0x1| | 0| | |
|-default-|board.mb.nb.pci_bus.io_space | | 0x0| | | | | |
+---------+-------------------------------+--+------+------+------+----+-----+----+

To find a port_space object, you can try below command in CLI (command line interface) window:
simics> list-objects iface = port_space -all

Trouble installing TIA portal: sqlcmd.exe is missing

I'm trying to install Siemens TIA portal, but get a setup error.
The setup states that "Some installed products need licenses", but shouldn't it just work with a trial license? It does give some error related to the missing of sqlcmd.exe.
The specific version I have is: TIA Portal STEP 7 Professional V15.1.
Setup screenshot: https://imgur.com/a/gNIXDHs
Log file:
2019.09.25 10:22:38 - Year.Month.Day - Log start
BUNDLE: SIMATIC TIA Portal STEP 7 Professional V15.1 - WinCC Advanced V15.1
SEBU Version: Advanced:TIA.rgs.RTM_15010000:V15.01.00.00_28.01.00.01
OS: Microsoft Windows 10 Pro Professional, 64-bit, build 18362
COMMANDLINE: C:\Program Files (x86)\Common Files\Siemens\Automation\Siemens Installer Assistant\406\SIA2.exe /starter /bundlepath=F:\TIA\InstData
INSTALLDIR: D:\Tia simulation
SETUPLANGUAGE: 1033
PRODUCTLANGUAGES: 1033
CHANGES: (+)DotNet_Updater;(o)CONFSQL2014EXP;(+)SYMLINK;(+)SYMLINKADD;(+)NCMGPRS64;(+)NCMFWL64;(+)OPC;(+)SECON;(o)S7DOS64;(o)S7DOS;(+)S7EDB;(+)Automation Software Updater;(+)TIA_Portal;(+)MU_Client;(+)HWCN-HWCOL;(+)SUPBASEH02;(+)SUPBASEH03;(+)SUPBASEH04;(+)SUPBASETO01;(+)SUPBASETO02;(+)SUPBASEWCF01;(+)TIAP_SIMATIC;(+)TIAP_SIMATIC32;(+)STEP7;(+)WinCC Runtime Advanced Simulator;(+)PTSymLib;(+)PTTouchInput;(+)HM_AllEditions;(+)HM_NoBasic;(+)UMC64;(+)LicenseManager;(+)STOP_ALMSERVICE;(o)AlmPanelPlugin64;(o)Prosave;(+)TIA_ES;(+)TIA_ES32;(+)WinCC_Tr_Mandatory;(+)WinCC_TrC_All;(+)WinCC_TrC_CAP;(+)CoRtHmiRTmDriver64;(+)HMIRTMV11;(+)RT_Adv_Sim;(+)TIACOMPCHECK;(+)TIA_Opns;(+)TIAADMIN;(+)AWB_SOFT;(+)AWB_ALM;(+)AWB_TIAUMAC;(+)MUSERVER
There are NO severe messages before the installation
10:22:38 | BEGIN. Execution started.
10:22:44 | OK RunningProcesses () Complete
10:22:44 | OK InventoryBefore (A) Complete
10:22:44 | OK Citamis (Before) Complete
10:23:12 | OK SecurityControlFeature (ReturnFeatures) Complete
10:23:12 | OK StartApplBefore (TIAP15_1::[INSTALLDIR]BIN\SIEMENS.AUTOMATION.OBJECTFRAME.FILESTORAGE.SERVER.EXE)Complete
10:23:12 | OK StartApplBefore (TIAP15_1::[INSTALLDIR]X64\BIN\SIEMENS.AUTOMATION.OBJECTFRAME.FILESTORAGE.SERVER.EXE)Complete
10:23:12 | OK ClassicCompPrepare () Complete
10:23:12 | OK ProductRegistration (Before) Complete
10:23:12 | OK ClassicCompAfterUninstall () Complete
10:23:12 | OK SetupUnit (DotNet_Updater) Complete Foreign AddLocal
10:23:12 | OK Checkpoint (E) Complete
10:23:14 |ERROR AdsWorkerClassicProduct::OnWorkerCompleted(): - ClassicProduct setup result: -1: File path: C:\Program Files (x86)\Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn\sqlcmd.exe is missing
10:23:14 | FAIL! SetupUnit (CONFSQL2014EXP) Failed ClassicProductReinstall
10:23:14 | FAIL! Checkpoint (E) Complete
10:23:14 |ERROR ExecutionEngine::StartNextStepAsync(): - Execution has been cancelled. Probably because of some process(es) running on the machine
10:23:14 | SKIP! SetupUnit (SYMLINK) Skip Foreign AddLocal
10:23:14 | SKIP! SetupUnit (SYMLINKADD) Skip Foreign AddLocal
10:23:14 | SKIP! SetupUnit (NCMGPRS64) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (NCMFWL64) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (OPC) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (SECON) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (S7DOS64) Skip ClassicComponentReinstall
10:23:14 | SKIP! SetupUnit (S7DOS) Skip ClassicComponentReinstall
10:23:14 | SKIP! SetupUnit (S7EDB) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (Automation Software Updater)Skip ClassicComponentAddLocal
10:23:14 | SKIP! SecurityControl (TIA_Portal) Skip
10:23:14 | SKIP! SetupUnit (TIA_Portal) Skip PlusMain AddLocal
10:23:14 | SKIP! SecurityControl (TIA_Portal) Skip
10:23:14 | SKIP! SetupUnit (MU_Client) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (HWCN-HWCOL) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASEH02) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASEH03) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASEH04) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASETO01) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASETO02) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (SUPBASEWCF01) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (TIAP_SIMATIC) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (TIAP_SIMATIC32) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (STEP7) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (WinCC Runtime Advanced Simulator)Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (PTSymLib) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (PTTouchInput) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SetupUnit (HM_AllEditions) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (HM_NoBasic) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SecurityControl (UMC64) Skip
10:23:14 | SKIP! SetupUnit (UMC64) Skip PlusMain AddLocal
10:23:14 | SKIP! SecurityControl (UMC64) Skip
10:23:14 | SKIP! SecurityControl (LicenseManager) Skip
10:23:14 | SKIP! SetupUnit (LicenseManager) Skip ClassicProductAddLocal
10:23:14 | SKIP! SecurityControl (LicenseManager) Skip
10:23:14 | SKIP! SetupUnit (STOP_ALMSERVICE) Skip Foreign AddLocal
10:23:14 | SKIP! SetupUnit (AlmPanelPlugin64) Skip ClassicComponentReinstall
10:23:14 | SKIP! SetupUnit (Prosave) Skip ClassicProductReinstall
10:23:14 | SKIP! SetupUnit (TIA_ES) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (TIA_ES32) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (WinCC_Tr_Mandatory) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (WinCC_TrC_All) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (WinCC_TrC_CAP) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (CoRtHmiRTmDriver64) Skip ClassicComponentAddLocal
10:23:14 | SKIP! SecurityControl (HMIRTMV11) Skip
10:23:14 | SKIP! SetupUnit (HMIRTMV11) Skip PlusMain AddLocal
10:23:14 | SKIP! SecurityControl (HMIRTMV11) Skip
10:23:14 | SKIP! SetupUnit (RT_Adv_Sim) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (TIACOMPCHECK) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SecurityControl (TIA_Opns) Skip
10:23:14 | SKIP! SetupUnit (TIA_Opns) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SecurityControl (TIA_Opns) Skip
10:23:14 | SKIP! SetupUnit (TIAADMIN) Skip PlusMain AddLocal
10:23:14 | SKIP! SetupUnit (AWB_SOFT) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (AWB_ALM) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (AWB_TIAUMAC) Skip PlusOptionalAddLocal
10:23:14 | SKIP! SetupUnit (MUSERVER) Skip PlusMain AddLocal
10:23:14 | SKIP! ClassicCompAfterInstall () Skip
10:23:14 | SKIP! ClassicCompRollback () Skip
10:23:14 | SKIP! HspInstall () Skip
10:23:14 | SKIP! MspInstall () Skip
10:23:14 | SKIP! StartApplAfter (TIAP15_1::[INSTALLDIR]BIN\INSTALLROOTCERTIFICATE.EXE)Skip
10:23:14 | SKIP! StartApplAfter (TIAP15_1::[INSTALLDIR]BIN\SIEMENS.AUTOMATION.GENERATOR.FRAMEWORK.EXE)Skip
10:23:14 | SKIP! StartApplAfter (TIAP15_1::C:\WINDOWS\SYSWOW64\S7EPATSX.EXE)Skip
10:23:14 | SKIP! StartApplAfter (TIAP15_1::C:\WINDOWS\SYSWOW64\S7EPAESX.EXE)Skip
10:23:14 | SKIP! StartApplAfter (TIAP15_1::[INSTALLDIR]BIN\SIEMENS.SIMATIC.HWCN.INTERPRETER.IG.GSD.GENERATOR.EXE)Skip
10:23:14 | SKIP! ProductRegistration (After) Skip
10:23:14 | SKIP! InventoryAfter (Z) Skip
10:23:14 | SKIP! LicenseTransfer (False) Skip
10:23:14 | SKIP! SetupReport () Skip
10:23:16 | OK SetupIsComplete () Complete
10:23:16 | OK Citamis (After) Complete
10:23:17 | OK SetupSummary () Complete
10:23:17 | END. Execution completed.
Intermediate ExitCode after execution queue : 1603 - Exit code means: Installation failed (InstallFailure)
Errors: 2
Warnings: 0
Thank you very much in advance.

Solved this by uninstalling Microsoft SQL server 2014, did this via the control panel. After that, I had to manually remove the Microsoft SQL server folder in Program Files and delete some registry keys.

Will selecting from pg_locks always return a result for itself?

SELECT relation::regclass, * FROM pg_locks ;
Results in the following:
relation | locktype | database | relation | page | tuple | virtualxid | transactionid | classid | objid | objsubid | virtualtransaction | pid | mode | granted | fastpath
----------+------------+----------+----------+------+-------+------------+---------------+---------+-------+----------+--------------------+------+-----------------+---------+----------
pg_locks | relation | 16397 | 11187 | | | | | | | | 76/111628 | 2652 | AccessShareLock | t | t
| virtualxid | | | | | 76/111628 | | | | | 76/111628 | 2652 | ExclusiveLock | t | t
(2 rows)
Can I assume that my query of pg_locks is itself what is causing the ExclusiveLock in that result?

sbt deduplicate: different file contents found in the following:

I'm aware this question has been asked (and answered) numerous times, but the existing answers do not address my case.
I am trying to build the DataStax spark-cassandra-connector (tag v1.5.0-M3) as per this tutorial
./sbt/sbt assembly
but I get the following errors
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
/Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
java.lang.RuntimeException: deduplicate: different file contents found in the following:
/Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
/Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
at sbtassembly.Assembly$.sbtassembly$Assembly$$applyStrategy$1(Assembly.scala:106)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:123)
at sbtassembly.Assembly$$anonfun$13.apply(Assembly.scala:120)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
at scala.collection.AbstractTraversable.flatMap(Traversable.scala:105)
at sbtassembly.Assembly$.applyStrategies(Assembly.scala:125)
at sbtassembly.Assembly$.x$1$lzycompute$1(Assembly.scala:25)
at sbtassembly.Assembly$.x$1$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$lzycompute$1(Assembly.scala:23)
at sbtassembly.Assembly$.stratMapping$1(Assembly.scala:23)
at sbtassembly.Assembly$.inputs$lzycompute$1(Assembly.scala:67)
at sbtassembly.Assembly$.inputs$1(Assembly.scala:57)
at sbtassembly.Assembly$.apply(Assembly.scala:83)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:226)
at sbtassembly.Assembly$$anonfun$assemblyTask$1.apply(Assembly.scala:223)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
at sbt.std.Transform$$anon$4.work(System.scala:63)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
at sbt.Execute.work(Execute.scala:235)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error] (spark-cassandra-connector-java/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/io.netty/netty/bundles/netty-3.8.0.Final.jar:META-INF/jboss-beans.xml
[error] /Users/me/.ivy2/cache/org.jboss.netty/netty/bundles/netty-3.2.2.Final.jar:META-INF/jboss-beans.xml
[error] (spark-cassandra-connector/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/me/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.5.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] /Users/me/.ivy2/cache/com.google.guava/guava/bundles/guava-16.0.1.jar:META-INF/maven/com.google.guava/guava/pom.properties
[error] Total time: 44 s, completed 10-Dec-2015 16:22:01
Any of the suggested answers I have read (such as this one) refer to changes required in the build.sbt file, but in my case I have no such file. In the project directory I have
.
├── BuildUtil.scala
├── CassandraSparkBuild.scala
├── Publish.scala
├── Settings.scala
├── SparkInstaller.scala
├── Versions.scala
├── build.properties
├── plugins.sbt
├── project
└── target
2 directories, 8 files
So I don't know which file(s) to edit and what edits are necessary. I am on OSX and have the following versions installed
Spark 1.5.2
Java 1.8.0_45
Scala 2.11.7
sbt 0.13.9
UPDATE
If I follow the instruction here and build for Scala 2.11 I get the following prompt
$ sbt -Dscala-2.11=true
[info] Loading project definition from /usr/local/share/spark-cassandra-connector/project
[info] Updating {file:/usr/local/share/spark-cassandra-connector/project/}spark-cassandra-connector-build...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
Using releases: https://oss.sonatype.org/service/local/staging/deploy/maven2 for releases
Using snapshots: https://oss.sonatype.org/content/repositories/snapshots for snapshots
Scala: 2.11.7
Scala Binary: 2.11
Java: target=1.7 user=1.8.0_45
[info] Set current project to root (in build file:/usr/local/share/spark-cassandra-connector/)
SBT   39dcadbe40e833b610357ce84811201f03d8f350  root 
Dependency Graph
Below is the result of using sbt-dependency-graph as per #AlexeyRomanovs suggestion
[info] default:project_2.10:0.1-SNAPSHOT [S]
[info] +-com.scalapenos:sbt-prompt:0.2.1
[info] | +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-git:0.6.4
[info] | +-org.eclipse.jgit:org.eclipse.jgit.pgm:3.3.2.201404171909-r
[info] | +-args4j:args4j:2.0.12
[info] | +-org.apache.commons:commons-compress:1.6
[info] | | +-org.tukaani:xz:1.4
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.archive:3.3.2.201404171909-r
[info] | | +-org.apache.commons:commons-compress:1.6
[info] | | | +-org.tukaani:xz:1.4
[info] | | |
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | | +-com.jcraft:jsch:0.1.50
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.osgi:org.osgi.core:4.3.1
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit.ui:3.3.2.201404171909-r
[info] | | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | | +-com.jcraft:jsch:0.1.50
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.eclipse.jgit:org.eclipse.jgit:3.3.2.201404171909-r
[info] | +-com.googlecode.javaewah:JavaEWAH:0.7.9
[info] | +-com.jcraft:jsch:0.1.50
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe.sbt:sbt-scalariform:1.3.0
[info] | +-org.scalariform:scalariform_2.10:0.1.4 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.scalastyle:scalastyle-sbt-plugin:0.6.0
[info] | +-org.scalastyle:scalastyle_2.10:0.6.0 [S]
[info] | +-com.danieltrinh:scalariform_2.10:0.1.5 [S]
[info] | +-com.typesafe:config:1.2.0
[info] |
[info] +-org.scoverage:sbt-scoverage:1.0.4
[info] | +-org.scoverage:scalac-scoverage-plugin_2.10:1.0.4 [S]
[info] |
[info] +-org.spark-packages:sbt-spark-package:0.2.3
[info] | +-com.eed3si9n:sbt-assembly:0.12.0
[info] | +-org.apache.directory.studio:org.apache.commons.codec:1.8
[info] | | +-commons-codec:commons-codec:1.8
[info] | |
[info] | +-org.scalaj:scalaj-http_2.10:1.1.4 [S]
[info] |
[info] +-org.typelevel:sbt-typelevel:0.3.1
[info] +-com.eed3si9n:sbt-buildinfo:0.3.2
[info] +-com.github.gseitz:sbt-release:0.8.4
[info] +-com.typesafe.sbt:sbt-pgp:0.8.2
[info] | +-com.jsuereth:gpg-library_2.10:0.8.2 [S]
[info] | | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | | | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | | +-commons-codec:commons-codec:1.8
[info] | | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | | |
[info] | | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | | +-commons-codec:commons-codec:1.8
[info] | | | | +-commons-logging:commons-logging:1.1.1
[info] | | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | | |
[info] | | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.bouncycastle:bcpg-jdk15on:1.49
[info] | | +-org.bouncycastle:bcprov-jdk15on:1.49
[info] | |
[info] | +-net.databinder:dispatch-http_2.10:0.8.10 [S]
[info] | +-net.databinder:dispatch-core_2.10:0.8.10 [S]
[info] | | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | | +-commons-codec:commons-codec:1.8
[info] | | | +-commons-logging:commons-logging:1.1.1
[info] | | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | | |
[info] | | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-net.databinder:dispatch-futures_2.10:0.8.10 [S]
[info] | | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.1.3 (evicted by: 4.2.6)
[info] | | +-commons-codec:commons-codec:1.4 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | | +-commons-codec:commons-codec:1.8
[info] | | +-commons-logging:commons-logging:1.1.1
[info] | | +-org.apache.httpcomponents:httpcore:4.1.4 (evicted by: 4.2.5)
[info] | | +-org.apache.httpcomponents:httpcore:4.2.5
[info] | |
[info] | +-org.apache.httpcomponents:httpclient:4.2.6
[info] | +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] | +-commons-codec:commons-codec:1.8
[info] | +-commons-logging:commons-logging:1.1.1
[info] | +-org.apache.httpcomponents:httpcore:4.2.5
[info] |
[info] +-com.typesafe:sbt-mima-plugin:0.1.6
[info] | +-com.typesafe:mima-reporter_2.10:0.1.6 [S]
[info] | +-com.typesafe:config:1.0.0 (evicted by: 1.2.0)
[info] | +-com.typesafe:config:1.2.0
[info] | +-com.typesafe:mima-core_2.10:0.1.6 [S]
[info] | | +-org.scala-lang:scala-compiler:2.10.2 [S]
[info] | | +-org.scala-lang:scala-reflect:2.10.2 [S]
[info] | |
[info] | +-org.scala-lang:scala-actors:2.10.2 [S]
[info] | +-org.scala-lang:scala-swing:2.10.2 [S]
[info] |
[info] +-net.virtual-void:sbt-dependency-graph:0.7.4 (evicted by: 0.8.0)
[info] +-net.virtual-void:sbt-dependency-graph:0.8.0
[info] | +-com.github.mdr:ascii-graphs_2.10:0.0.3 [S]
[info] |
[info] +-org.xerial.sbt:sbt-sonatype:0.2.1
[info] +-org.apache.httpcomponents:httpclient:4.2.6
[info] +-commons-codec:commons-codec:1.6 (evicted by: 1.8)
[info] +-commons-codec:commons-codec:1.8
[info] +-commons-logging:commons-logging:1.1.1
[info] +-org.apache.httpcomponents:httpcore:4.2.5
[info]

If you look at the SBT tutorial, you'll find that project/*.scala files are equivalent to *.sbt files in the root directory. So that's where you need to make the changes (specifically, in CassandraSparkBuild.scala).
The problem is that there are two versions of Netty in transitive dependencies and SBT doesn't notice this because the group IDs are different: org.jboss.netty for 3.2.2 and io.netty for 3.8. Find where they come from (using e.g. https://github.com/jrudolph/sbt-dependency-graph) and exclude the older version (and hope nothing breaks).

Scalding on EMR: Hadoop job fails with NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;

Basically I need to run a scalding job on EMR. The same job runs perfectly fine on local hadoop on my macbook, but fails on Hadoop on EMR.
I am trying hard to get help for this issue in the cascading-user and scala-user groups as well, and haven't been able to. So far I haven't made much progress after trying various changes in the past couple days.
Here is the error before I delve into the details:
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.ArrowAssoc(Ljava/lang/Object;)Ljava/lang/Object;
at com.aggregation.job.DataAggregation$.(DataAggregation.scala:30)
at com.aggregation.job.DataAggregation$.(DataAggregation.scala)
at com.aggregation.job.DataAggregation.main(DataAggregation.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
They said it might be a scala conflict with the binaries, but I couldn't see anything obvious. It would be great if someone could help with figuring it out.
Environment:
Amazon EMR AMI: 3.8.0 (which includes Scala 2.11.1, Hadoop 2.4.0, Java 1.7.0_76 - AMI details here)
Application environment: Scalding 0.15.0, Scala 2.11.1, Java 1.7.0_80, Hadoop 2.4.0
I have tried various changes to configuration and even manually installing a higher scala version in EMR, but so far the error is not going away.
Please help! Thank you.
L
Setup
build.sbt:
scalaVersion := "2.11.1"
ivyScala := ivyScala.value map {
_.copy(overrideScalaVersion = true)
}
dependencies.sbt:
import AssemblyKeys._
val hadoopVersion = "2.4.0"
val scaldingVersion = "0.15.0"
libraryDependencies ++= Seq(
"com.twitter" %% "scalding-core" % scaldingVersion,
"com.twitter" %% "scalding-json" % scaldingVersion,
"com.twitter" %% "scalding-jdbc" % scaldingVersion,
"com.github.nscala-time" %% "nscala-time" % "2.0.0",
"org.apache.hadoop" % "hadoop-common" % hadoopVersion % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % hadoopVersion % "provided"
)
excludedJars in assembly <<= (fullClasspath in assembly) map { cp =>
val excludes = Set(
"jsp-api-2.1-6.1.14.jar",
"jsp-2.1-6.1.14.jar",
"jasper-compiler-5.5.12.jar",
"minlog-1.2.jar", // Otherwise causes conflicts with Kyro (which bundles it)
"janino-2.5.16.jar", // Janino includes a broken signature, and is not needed anyway
"commons-beanutils-core-1.8.0.jar", // Clash with each other and with commons-collections
"commons-beanutils-1.7.0.jar", // "
"hadoop-core-1.2.1.jar", // Provided by Amazon EMR. Delete this line if you're not on EMR
"hadoop-tools-1.2.1.jar" // "
)
cp filter { jar => excludes(jar.data.getName) }
}
resolvers ++= Seq(
"Conjars repo" at "http://conjars.org/repo"
)
assembly.sbt
import AssemblyKeys._
assemblySettings
mergeStrategy in assembly := Merge.mergeStrategy
project/build.properties:
sbt.version=0.13.1
project/assembly.sbt:
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.0")
Finally this is the dependency tree using the sbt-dependency-graph plugin to see if I've got the right versions. Sorry, it's quite long, if I should be displaying this info any other way, please recommend suggestions.
sbt dependency tree:
[info] com.abc.aggregator_2.11:0.1-20150628T184441 [S]
[info] +-com.github.nscala-time:nscala-time_2.11:2.0.0 [S]
[info] | +-joda-time:joda-time:2.7
[info] | +-org.joda:joda-convert:1.2
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-jdbc_2.11:0.15.0 [S]
[info] | +-cascading:cascading-jdbc-core:2.6.0
[info] | | +-com.google.guava:guava:15.0
[info] | |
[info] | +-cascading:cascading-jdbc-mysql:2.6.0
[info] | | +-cascading:cascading-jdbc-core:2.6.0
[info] | | | +-com.google.guava:guava:15.0
[info] | | |
[info] | | +-com.google.guava:guava:15.0
[info] | | +-mysql:mysql-connector-java:5.1.25
[info] | |
[info] | +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-com.twitter:scalding-json_2.11:0.15.0 [S]
[info] +-com.fasterxml.jackson.module:jackson-module-scala_2.11:2.4.2 [S]
[info] | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | +-com.fasterxml.jackson.core:jackson-databind:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.0 (evicted by: 2.4.2)
[info] | | +-com.fasterxml.jackson.core:jackson-annotations:2.4.2
[info] | | +-com.fasterxml.jackson.core:jackson-core:2.4.2
[info] | |
[info] | +-com.google.code.findbugs:jsr305:2.0.1
[info] | +-com.google.guava:guava:15.0
[info] | +-com.thoughtworks.paranamer:paranamer:2.6
[info] | +-org.scala-lang:scala-reflect:2.11.2 [S]
[info] |
[info] +-com.twitter:scalding-core_2.11:0.15.0 [S]
[info] | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-cascading:cascading-local:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | | +-org.codehaus.janino:janino:2.7.5
[info] | | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | | |
[info] | | | +-riffle:riffle:0.1-dev
[info] | | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | | |
[info] | | +-com.google.guava:guava:14.0.1 (evicted by: 15.0)
[info] | | +-com.google.guava:guava:15.0
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | | +-org.slf4j:slf4j-api:1.7.2 (evicted by: 1.6.6)
[info] | |
[info] | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | |
[info] | +-com.twitter:bijection-core_2.11:0.8.0 [S]
[info] | +-com.twitter:chill-algebird_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:algebird-core_2.11:0.10.0 (evicted by: 0.10.1)
[info] | | +-com.twitter:algebird-core_2.11:0.10.1 [S]
[info] | | | +-com.googlecode.javaewah:JavaEWAH:0.6.6
[info] | | |
[info] | | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill-hadoop:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-org.slf4j:slf4j-api:1.6.6
[info] | |
[info] | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:chill_2.11:0.6.0 [S]
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | | +-org.ow2.asm:asm:4.0
[info] | | | |
[info] | | | +-org.objenesis:objenesis:1.2
[info] | | |
[info] | | +-com.twitter:chill-java:0.6.0
[info] | | +-com.esotericsoftware.kryo:kryo:2.21
[info] | | +-com.esotericsoftware.minlog:minlog:1.2
[info] | | +-com.esotericsoftware.reflectasm:reflectasm:1.07
[info] | | | +-org.ow2.asm:asm:4.0
[info] | | |
[info] | | +-org.objenesis:objenesis:1.2
[info] | |
[info] | +-com.twitter:maple:0.15.0
[info] | | +-cascading:cascading-hadoop:2.6.1
[info] | | +-cascading:cascading-core:2.6.1
[info] | | +-org.codehaus.janino:janino:2.7.5
[info] | | | +-org.codehaus.janino:commons-compiler:2.7.5
[info] | | |
[info] | | +-riffle:riffle:0.1-dev
[info] | | +-thirdparty:jgrapht-jdk1.6:0.8.1
[info] | |
[info] | +-com.twitter:scalding-args_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-date_2.11:0.15.0 [S]
[info] | +-com.twitter:scalding-serialization_2.11:0.15.0 [S]
[info] | +-org.slf4j:slf4j-api:1.6.6
[info] |
[info] +-org.json4s:json4s-native_2.11:3.2.11 [S]
[info] +-org.json4s:json4s-core_2.11:3.2.11 [S]
[info] +-com.thoughtworks.paranamer:paranamer:2.6
[info] +-org.json4s:json4s-ast_2.11:3.2.11 [S]
[info] +-org.scala-lang:scalap:2.11.0
[info] +-org.scala-lang:scala-compiler:2.11.1 [S]
[info] +-org.scala-lang.modules:scala-parser-combinators_2.11:1.0.1 [S]
[info] +-org.scala-lang.modules:scala-xml_2.11:1.0.2 [S]
[info] +-org.scala-lang:scala-reflect:2.11.1 (evicted by: 2.11.2)
[info] +-org.scala-lang:scala-reflect:2.11.2 [S]
[info]
[success] Total time: 12 s, completed Jun 28, 2015 11:44:54 AM
Further info as requested:
I build the fat jar using 'sbt assembly', and currently I'm using the AWS console with a "Custom JAR" step to test this out before automating the process.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt
UPDATE:
I was able to get past the above error by providing the HADOOP_CLASSPATH pointing to the scala 2.11.1 jars, while excluding the same from the sbt assembly step. This was passed in using hadoop-user-env.sh and seemed to work for the master node. However once it got to the mapper step it once again failed with another Scala error. Now I am stuck on this step.
Assuming this is because the mappers and reducers aren't seeing the HADOOP_CLASSPATH update, I tried including the -libjars argument pointing to the scala jar files on hadoop master itself. But this (below) doesn't seem to be working.
JAR location: s3://path/to/jar/data-aggregator-0.1.jar
Arguments: com.abc.aggregation.job.DataAggregation -libjars /usr/share/scala/lib/scala-library.jar,/usr/share/scala/lib/scala-reflect.jar --hdfs --input s3n://path/to/input/data/file.json --output s3n://path/to/input/data/file.txt

Fixed. So it does happen that there were multiple scala jars in the EMR instances, and they weren't coming from my application jar.
The 2.10 jar was hiding in /usr/share/aws/emr/emrfs/lib apart from the installed location for the 2.11 binaries under /usr/share/scala. So I got rid of the 2.10 jar in all instances of the cluster, and my job completed successfully. Now I will create a bootstrap action for this.
$ sudo find / -name "scala-library-2.10.*.jar" -exec rm -rf {} \;
FYI, these are the paths it was present under:
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.11.*.jar"
/home/hadoop/.versions/hbase-0.94.18/lib/scala-library-2.11.0.jar
/usr/share/doc/scala/api/jars/scala-library-2.11.1-javadoc.jar
[ec2-user#ip-172-31-72-130 ~]$ sudo find / -name "scala-library-2.10.*.jar"
/usr/share/aws/emr/emrfs/lib/scala-library-2.10.5.jar

After struggling with a similar issue..... which is clearly a version mismatch between jar dependencies. I fixed it by first logging into the EMR master, then opening spark-shell reveals the exact scala version that is being used. Now take that to configure your own dependencies in the pom file and it should work.