Skip to content

BIGTOP-3992：Missing distcp package for hdfs in hive #1167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lvkaihua wants to merge 5 commits into
base: master
Choose a base branch
from
Open

BIGTOP-3992：Missing distcp package for hdfs in hive #1167

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0aa1116
Update hive.spec
lvkaihua Aug 21, 2023
b9349bf
Update rules
lvkaihua Aug 21, 2023
f02add8
Update rules
lvkaihua Sep 19, 2023
0439c18
Update rules
lvkaihua Sep 19, 2023
0f2d5ec
Update rules
lvkaihua Sep 20, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions bigtop-packages/src/deb/hive/rules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ override_dh_auto_install: server2 metastore hcatalog-server webhcat-server
ln -s /usr/lib/hbase/hbase-common.jar /usr/lib/hbase/hbase-client.jar /usr/lib/hbase/hbase-hadoop-compat.jar /usr/lib/hbase/hbase-hadoop2-compat.jar debian/tmp/usr/lib/hive/lib
ln -s /usr/lib/hbase/hbase-procedure.jar /usr/lib/hbase/hbase-protocol.jar /usr/lib/hbase/hbase-server.jar debian/tmp/usr/lib/hive/lib/
ln -s /usr/lib/zookeeper/zookeeper.jar debian/tmp/usr/lib/hive/lib
ln -s /usr/lib/hadoop//tools/lib/hadoop-distcp*.jar debian/tmp/usr/lib/hive/lib/
# Workaround for BIGTOP-583
rm -f debian/tmp/usr/lib/hive/lib/slf4j-log4j12-*.jar
bash debian/build-hive-install-file.sh >> debian/hive.install
2 changes: 2 additions & 0 deletions bigtop-packages/src/rpm/hive/SPECS/hive.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@

%define usr_lib_zookeeper %{parent_dir}/usr/lib/zookeeper
%define usr_lib_hbase %{parent_dir}/usr/lib/hbase
%define usr_lib_hadoop %{parent_dir}/usr/lib/hadoop

%define bin_dir %{parent_dir}/%{_bindir}
%define man_dir %{parent_dir}/%{_mandir}
Expand Down Expand Up @@ -295,6 +296,7 @@ cp $RPM_SOURCE_DIR/hive-site.xml .
%__ln_s %{usr_lib_zookeeper}/zookeeper.jar $RPM_BUILD_ROOT/%{usr_lib_hive}/lib/
%__ln_s %{usr_lib_hbase}/hbase-common.jar %{usr_lib_hbase}/hbase-client.jar %{usr_lib_hbase}/hbase-hadoop-compat.jar %{usr_lib_hbase}/hbase-hadoop2-compat.jar $RPM_BUILD_ROOT/%{usr_lib_hive}/lib/
%__ln_s %{usr_lib_hbase}/hbase-procedure.jar %{usr_lib_hbase}/hbase-protocol.jar %{usr_lib_hbase}/hbase-server.jar $RPM_BUILD_ROOT/%{usr_lib_hive}/lib/
%__ln_s %{usr_lib_hadoop}/tools/lib/hadoop-distcp-*.jar $RPM_BUILD_ROOT/%{usr_lib_hive}/lib/
Copy link
Member

@iwasakims iwasakims Sep 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the same issue with RPM on Rocky Linux 8.

[root@c573e05810ea /]# cat /etc/os-release | head -n 2
NAME="Rocky Linux"
VERSION="8.5 (Green Obsidian)"

[root@c573e05810ea /]# ls -l /usr/lib/hive/lib | grep distcp
lrwxrwxrwx. 1 root root       45 Sep 19 11:55 hadoop-distcp-*.jar -> /usr/lib/hadoop/tools/lib/hadoop-distcp-*.jar

Copy link
Contributor Author

@lvkaihua lvkaihua Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know much about Rocky Linux, does it use rpm? I had no issues with Centos before. If there are any issues with Rocky Linux, I will test this system again using rpm.
微信截图_20230920110003

Copy link
Member

@iwasakims iwasakims Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops. I meant RPM on Rocky Linux 8.

Copy link
Contributor Author

@lvkaihua lvkaihua Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

微信截图_20230920113638
I tested on Rocky Linux, but there seems to be no problem

Copy link
Contributor Author

@lvkaihua lvkaihua Sep 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it's because you need to install Hadoop's rpm first?


# Workaround for BIGTOP-583
%__rm -f $RPM_BUILD_ROOT/%{usr_lib_hive}/lib/slf4j-log4j12-*.jar
Expand Down