hadoop - Retrieve only file name into a table in hive -
i need retrieve filename.txt linux path , insert filename table column in hive. possible retrieve file name path , insert hive table using virtual columns? please advice!
e.g. of path /home/usr/path/filename.txt , insert filename table. create table t( name string);
thanks!
if want run against hdfs - command - awk -f "/" '{print $nf}'
file name.
[cloudera@quickstart ~]$ hadoop fs -ls /user/cloudera/departments|awk -f "/" '{print $nf}'|egrep -v 'found|_success' part-m-00000 part-m-00001 [cloudera@quickstart ~]$
if want run against local file system - command - ls -1
give file name. can use awk -f "/" '{print $nf}'
you can create shell script as: (uncomment hive
statements)
#!/bin/sh files=`hadoop fs -ls /user/cloudera/departments|awk -f "/" '{print $nf}'|egrep -v 'found|_success'` file in $files #hive -e "insert table t(name) values (\"$file\");" echo "insert table t(name) values (\"$file\");" done
should instert in hive table:
[cloudera@quickstart ~]$ ./test.sh insert table t(name) values ("part-m-00000"); insert table t(name) values ("part-m-00001");
Comments
Post a Comment