What is misconfigured with my maven project? Not picking up dependencies or class structure - scala

Question

There appears to be something misconfigured with my project that's causing maven to miss the dependencies in my base pom.xml. When I try to build a module with intellij's build options, or build my project with mvn clean package, I get an error that 'object apache is not a member of package org'. My pom file has the following:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
<exclusions>
...
</exclusions>
</dependency>
I have confirmed that the spark-sql_2.11 (my version) jar exists in my local .m2 repository, and that my settings are configured to import maven projects automatically.
When I remove references to org.apache._ the jar builds successfully with both mvn clean package and intellij's build module menu option, however the two jars do not function the same. The jar created from building the module through intellij (/out/artifacts/[project]_jar/[project].jar) works fine with spark-submit, however using the jar created from mvn clean package (/target/[project]-test-SNAPSHOT.jar) throws an error 'No main class set in jar' despite the fact that it appears there is, 'jar tvf ' shows this class, and MANIFEST.MF has it correctly listed as the main class.
It almost seems like there's a disconnect between my scala class and the maven project, but I can't understand where. Any tips on how to diagnose why this isn't working would be greatly appreciated, and I can absolutely provide additional details if necessary. Thanks in advance!
Edit: adding full contents of pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>basepom</artifactId>
<groupId>com.cdtk.maven.basepom</groupId>
<version>1.0-SNAPSHOT</version>
<relativePath />
</parent>
<groupId>com.[company].testproject2</groupId>
<artifactId>swtest2</artifactId>
<version>test-SNAPSHOT</version>
<packaging>jar</packaging>
<description>
This parent pom holds all the necessary version
and configuration information for all submodules.
</description>
<properties>
<required.maven.version>3.5.2</required.maven.version>
<upper.maven.version>3.5.999</upper.maven.version>
<required.java.update>20</required.java.update>
<scala.major.version>2</scala.major.version>
<scala.minor.version>11</scala.minor.version>
<scala.patch.version>0</scala.patch.version>
<spark.version>2.3.0</spark.version>
</properties>
<dependencyManagement>
<dependencies>
<!-- =================================== -->
<!-- Core Spark BOM -->
<!-- =================================== -->
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
<exclusions>
<exclusion>
<artifactId>commons-beanutils-core</artifactId>
<groupId>commons-beanutils</groupId>
</exclusion>
<exclusion>
<artifactId>jersey-server</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>javax.inject</artifactId>
<groupId>javax.inject</groupId>
</exclusion>
<exclusion>
<artifactId>javax.ws.rs-api</artifactId>
<groupId>javax.ws.rs</groupId>
</exclusion>
<exclusion>
<artifactId>stax-api</artifactId>
<groupId>javax.xml.stream</groupId>
</exclusion>
<exclusion>
<artifactId>aopalliance</artifactId>
<groupId>aopalliance</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
<exclusion>
<artifactId>paranamer</artifactId>
<groupId>com.thoughtworks.paranamer</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<version>1.60</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>2.6.5</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>1.1.2.6</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<version>3.4.1</version>
</dependency>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
</dependency>
<dependency>
<groupId>org.scala-lang.modules</groupId>
<artifactId>scala-parser-combinators_${scala.major.version}.${scala.minor.version}</artifactId>
<version>1.0.4</version>
</dependency>
<dependency>
<groupId>org.codehaus.janino</groupId>
<artifactId>commons-compiler</artifactId>
<version>3.0.8</version>
</dependency>
<dependency>
<groupId>org.glassfish.hk2.external</groupId>
<artifactId>javax.inject</artifactId>
<version>2.5.0-b32</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-common</artifactId>
<version>2.25.1</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>3.9.9.Final</version>
</dependency>
<dependency>
<groupId>javax.activation</groupId>
<artifactId>activation</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-client</artifactId>
<version>2.25.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>commons-net</groupId>
<artifactId>commons-net</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-xc</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-jaxrs</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.13</version>
</dependency>
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-reflect</artifactId>
<version>${scala.major.version}.${scala.minor.version}.${scala.patch.version}</version>
</dependency>
<!-- =================================== -->
<!-- Top level dependencies -->
<!-- =================================== -->
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.oracle</groupId>
<artifactId>ojdbc6</artifactId>
<version>11.2.0.3</version>
</dependency>
<dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-avro_${scala.major.version}.${scala.minor.version}</artifactId>
<version>4.0.0</version>
</dependency>
<!-- =================================== -->
<!-- Test dependencies -->
<!-- =================================== -->
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
<version>3.0.5</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>2.0.0</version>
<scope>test</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.major.version}.${scala.minor.version}</artifactId>
<exclusions>
<exclusion>
<artifactId>scala-library</artifactId>
<groupId>org.scala-lang</groupId>
</exclusion>
<exclusion>
<artifactId>scala-xml_${scala.major.version}.${scala.minor.version}</artifactId>
<groupId>org.scala-lang.modules</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
</plugin>
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>2.0.0</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaCompatVersion>${scala.major.version}.${scala.minor.version}</scalaCompatVersion>
</configuration>
</plugin>
<plugin>
<groupId>org.basepom.maven</groupId>
<artifactId>duplicate-finder-maven-plugin</artifactId>
<configuration>
<ignoredClassPatterns combine.self="append">
<ignoredClassPattern>org.apache.spark.unused.UnusedStubClass</ignoredClassPattern>
</ignoredClassPatterns>
<ignoredResourcePatterns combine.self="append">
<!-- mercury internal file (should provide link to code that proves its ok to have more than one on the classpath) -->
<ignoredResource>overrides.conf</ignoredResource>
<ignoredResource>git.properties</ignoredResource>
</ignoredResourcePatterns>
</configuration>
</plugin>
</plugins>
</build>
</project>

Answer 1

You should add the spark-core and spark-sql dependencies in your child pom or in current pom, if not add it as below
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.major.version}.${scala.minor.version}</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.major.version}.${scala.minor.version}</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
Take a look at Difference between dependencies and dependency management
For the main class not found
Add the maven-assembly-plugin plugin to the child pom as

Answer 2

You can add maven assembly jar to build a fat jar. Add the below plugin and update the name of your main class.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.6</version>
<executions>
<execution>
<id>jar-with-dependencies</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>***com.abc.mainclass***</mainClass>
</manifest>
</archive>
</configuration>
</execution>
</executions>
</plugin>