How to enable YARN MapReduce jobs with ofs:// output?

[silvanias]

Hey Ozone team,

We're working on integrating Apache Ozone with our existing Hadoop YARN cluster. We've made good progress getting basic Ozone operations working + distcp between hdfs and ozone, but YARN jobs are failing with a Ratis network error that we can't figure out.

Any guidance would be hugely appreciated! Details below:

Environment

• Ozone: 2.0.0 (3 OMs in HA, Kerberos enabled)
• Hadoop: 3.3 (HDP)
• Setup: Separate HDFS and Ozone clusters (non-colocated)

Goal

Run MapReduce jobs that read from HDFS and write to Ozone:

hadoop jar hadoop-mapreduce-examples-*.jar wordcount \
  hdfs://dev3/user/test/input.txt \
  ofs://dev3/user/test/output

Current Setup

Client Environment (before job submission):

cp /etc/ozone/ozone-site.xml /etc/hadoop/conf.client/
export HADOOP_CONF_DIR=/etc/hadoop/conf.client/
export YARN_CONF_DIR=$HADOOP_CONF_DIR
export YARN_USER_CLASSPATH_FIRST=true
export HADOOP_CLASSPATH=/usr/share/ozone/share/ozone/lib/*.jar

All YARN Nodes (ResourceManager + NodeManagers):

JARs copied to /usr/hdp/hadoop-3.3/share/hadoop/common/lib/:

• ozone-filesystem-hadoop3-*.jar
• ozone-client-*.jar
• hdds-common-*.jar
• hdds-interface-*.jar
• hdds-hadoop-dependency-client-*.jar

Configuration:

• ozone-site.xml copied to /etc/hadoop/conf/
• Added to core-site.xml:

<property>
  <name>fs.ofs.impl</name>
  <value>org.apache.hadoop.fs.ozone.RootedOzoneFileSystem</value>
</property>

Job Submission:

hadoop jar hadoop-mapreduce-examples-*.jar wordcount \
  -Dmapreduce.job.user.classpath.first=true \
  hdfs://dev3/user/test/input.txt \
  ofs://dev3/user/test/output

Error

Job fails immediately with:

Client output:

Container exited with a non-zero exit code 1.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).

Application Master syslog (from YARN UI):

java.io.IOException: java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE:
Network closed for unknown reason

Questions

1. What JARs are required on YARN nodes (ResourceManager and NodeManagers) for ofs:// support?

   • Are we missing any critical JARs?

2. What configuration files are needed on YARN nodes?

   • Is ozone-site.xml in /etc/hadoop/conf/ sufficient?
   • Do we need additional properties in core-site.xml or yarn-site.xml?

3. What causes the "Network closed for unknown reason" Ratis error?

   • Is this a network connectivity issue between YARN containers and Ozone?
   • What ports/protocols need to be accessible from NodeManagers to Ozone cluster?

Additional Info

• Basic Ozone operations work from client: hadoop fs -ls ofs://dev3/, put, get, cat
• HDFS-to-HDFS MapReduce jobs work fine
• HDFS-to-Ozone Distcp works fine
• Running the map-reduce job locally on the node works fine
• Kerberos authentication successful
• All Ozone services healthy (OMs, SCMs, DataNodes)

[adoroszlai]

Thanks @silvanias for trying Ozone.

For Hadoop, YARN and most ecosystem applications, please use only ozone-filesystem-hadoop3-2.0.0.jar. It is a fat jar, contains all Ozone client components, as well as dependencies for using Ozone in such an environment. Adding other Ozone jars will likely cause problems due to duplicate classes.

For using OFS from FileContext API, the following config is required in core-site.xml:

<property>
  <name>fs.AbstractFileSystem.ofs.impl</name>
  <value>org.apache.hadoop.fs.ozone.RootedOzFs</value>
</property>

Connection to OM (9862), SCM (9863) and Ozone Datanodes (9855/9858/9859) are minimally required. I hope I have not missed anything.

[mdellabitta]

Not sure if jumping on someone else's post is bad etiquette, but I'm encountering similar problems.

Accessing ozone from the hadoop fs cli works fine. When I go to launch a job, YARN can see the input directory and fails if the output directory exists, so it seems good to go with config and jars. But the mapper dies right away, saying it doesn't know the ofs filesystem.

This is a proof of concept Ozone 2.0.0/Hadoop 3.4.2 cluster with no HDFS, fs.defaultFS is set to ofs://$OM_NODE.

Anything I should be digging into?

[devabhishekpal]

I am closing this discussion as @adoroszlai's answer was accepted by @silvanias.

@mdellabitta could you raise this as a different discussion to get better visibility on the issue?