经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 大数据/云/AI » MapReduce » 查看文章
MapReduce报错Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio
来源：cnblogs　　作者：zhuzhu&you　　时间：2020/12/8 8:44:12　　对本文有异议

在使用MapReduce的小测试的时候，Driver类启动，或报下面的错误共有两种解决方案

  1. log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
  2. log4j:WARN Please initialize the log4j system properly.
  3. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  4. Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
  5.     at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
  6.     at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:609)
  7.     at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
  8.     at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskChecker.java:187)
  9.     at org.apache.hadoop.util.DiskChecker.checkDirAccess(DiskChecker.java:174)
  10.     at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:108)
  11.     at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:285)
  12.     at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:344)
  13.     at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
  14.     at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
  15.     at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115)
  16.     at org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:125)
  17.     at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163)
  18.     at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731)
  19.     at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
  20.     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
  21.     at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
  22.     at java.security.AccessController.doPrivileged(Native Method)
  23.     at javax.security.auth.Subject.doAs(Subject.java:422)
  24.     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
  25.     at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
  26.     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
  27.     at com.runner.WordCountRunner.main(WordCountRunner.java:45)
  28.  
  29. Process finished with exit code 1

出现原因：在新版本的windows系统中，会取消部分文件，某些功能无法支持。本地的NativeIO无法写入，

第一种：我们需要再写一个NativeIO的类，放入代码片段的包中；

 

NativeIO的代码如下，请自行copy,package自行修改

 

  1. /**
  2.  * Licensed to the Apache Software Foundation (ASF) under one
  3.  * or more contributor license agreements.  See the NOTICE file
  4.  * distributed with this work for additional information
  5.  * regarding copyright ownership.  The ASF licenses this file
  6.  * to you under the Apache License, Version 2.0 (the
  7.  * "License"); you may not use this file except in compliance
  8.  * with the License.  You may obtain a copy of the License at
  9.  *
  10.  *     http://www.apache.org/licenses/LICENSE-2.0
  11.  *
  12.  * Unless required by applicable law or agreed to in writing, software
  13.  * distributed under the License is distributed on an "AS IS" BASIS,
  14.  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  15.  * See the License for the specific language governing permissions and
  16.  * limitations under the License.
  17.  */
  18. package org.apache.hadoop.io.nativeio;
  19.  
  20. import java.io.File;
  21. import java.io.FileDescriptor;
  22. import java.io.FileInputStream;
  23. import java.io.FileOutputStream;
  24. import java.io.IOException;
  25. import java.io.RandomAccessFile;
  26. import java.lang.reflect.Field;
  27. import java.nio.ByteBuffer;
  28. import java.nio.MappedByteBuffer;
  29. import java.nio.channels.FileChannel;
  30. import java.util.Map;
  31. import java.util.concurrent.ConcurrentHashMap;
  32.  
  33. import org.apache.hadoop.classification.InterfaceAudience;
  34. import org.apache.hadoop.classification.InterfaceStability;
  35. import org.apache.hadoop.conf.Configuration;
  36. import org.apache.hadoop.fs.CommonConfigurationKeys;
  37. import org.apache.hadoop.fs.HardLink;
  38. import org.apache.hadoop.io.IOUtils;
  39. import org.apache.hadoop.io.SecureIOUtils.AlreadyExistsException;
  40. import org.apache.hadoop.util.NativeCodeLoader;
  41. import org.apache.hadoop.util.Shell;
  42. import org.apache.hadoop.util.PerformanceAdvisory;
  43. import org.apache.commons.logging.Log;
  44. import org.apache.commons.logging.LogFactory;
  45.  
  46. import sun.misc.Unsafe;
  47.  
  48. import com.google.common.annotations.VisibleForTesting;
  49.  
  50. /**
  51.  * JNI wrappers for various native IO-related calls not available in Java.
  52.  * These functions should generally be used alongside a fallback to another
  53.  * more portable mechanism.
  54.  */
  55. @InterfaceAudience.Private
  56. @InterfaceStability.Unstable
  57. public class NativeIO {
  58.     public static class POSIX {
  59.         // Flags for open() call from bits/fcntl.h
  60.         public static final int O_RDONLY   =    00;
  61.         public static final int O_WRONLY   =    01;
  62.         public static final int O_RDWR     =    02;
  63.         public static final int O_CREAT    =  0100;
  64.         public static final int O_EXCL     =  0200;
  65.         public static final int O_NOCTTY   =  0400;
  66.         public static final int O_TRUNC    = 01000;
  67.         public static final int O_APPEND   = 02000;
  68.         public static final int O_NONBLOCK = 04000;
  69.         public static final int O_SYNC   =  010000;
  70.         public static final int O_ASYNC  =  020000;
  71.         public static final int O_FSYNC = O_SYNC;
  72.         public static final int O_NDELAY = O_NONBLOCK;
  73.  
  74.         // Flags for posix_fadvise() from bits/fcntl.h
  75.         /* No further special treatment.  */
  76.         public static final int POSIX_FADV_NORMAL = 0;
  77.         /* Expect random page references.  */
  78.         public static final int POSIX_FADV_RANDOM = 1;
  79.         /* Expect sequential page references.  */
  80.         public static final int POSIX_FADV_SEQUENTIAL = 2;
  81.         /* Will need these pages.  */
  82.         public static final int POSIX_FADV_WILLNEED = 3;
  83.         /* Don't need these pages.  */
  84.         public static final int POSIX_FADV_DONTNEED = 4;
  85.         /* Data will be accessed once.  */
  86.         public static final int POSIX_FADV_NOREUSE = 5;
  87.  
  88.  
  89.         /* Wait upon writeout of all pages
  90.            in the range before performing the
  91.            write.  */
  92.         public static final int SYNC_FILE_RANGE_WAIT_BEFORE = 1;
  93.         /* Initiate writeout of all those
  94.            dirty pages in the range which are
  95.            not presently under writeback.  */
  96.         public static final int SYNC_FILE_RANGE_WRITE = 2;
  97.  
  98.         /* Wait upon writeout of all pages in
  99.            the range after performing the
  100.            write.  */
  101.         public static final int SYNC_FILE_RANGE_WAIT_AFTER = 4;
  102.  
  103.         private static final Log LOG = LogFactory.getLog(NativeIO.class);
  104.  
  105.         private static boolean nativeLoaded = false;
  106.         private static boolean fadvisePossible = true;
  107.         private static boolean syncFileRangePossible = true;
  108.  
  109.         static final String WORKAROUND_NON_THREADSAFE_CALLS_KEY =
  110.                 "hadoop.workaround.non.threadsafe.getpwuid";
  111.         static final boolean WORKAROUND_NON_THREADSAFE_CALLS_DEFAULT = true;
  112.  
  113.         private static long cacheTimeout = -1;
  114.  
  115.         private static CacheManipulator cacheManipulator = new CacheManipulator();
  116.  
  117.         public static CacheManipulator getCacheManipulator() {
  118.             return cacheManipulator;
  119.         }
  120.  
  121.         public static void setCacheManipulator(CacheManipulator cacheManipulator) {
  122.             POSIX.cacheManipulator = cacheManipulator;
  123.         }
  124.  
  125.         /**
  126.          * Used to manipulate the operating system cache.
  127.          */
  128.         @VisibleForTesting
  129.         public static class CacheManipulator {
  130.             public void mlock(String identifier, ByteBuffer buffer,
  131.                               long len) throws IOException {
  132.                 POSIX.mlock(buffer, len);
  133.             }
  134.  
  135.             public long getMemlockLimit() {
  136.                 return NativeIO.getMemlockLimit();
  137.             }
  138.  
  139.             public long getOperatingSystemPageSize() {
  140.                 return NativeIO.getOperatingSystemPageSize();
  141.             }
  142.  
  143.             public void posixFadviseIfPossible(String identifier,
  144.                                                FileDescriptor fd, long offset, long len, int flags)
  145.                     throws NativeIOException {
  146.                 POSIX.posixFadviseIfPossible(identifier, fd, offset,
  147.                         len, flags);
  148.             }
  149.  
  150.             public boolean verifyCanMlock() {
  151.                 return NativeIO.isAvailable();
  152.             }
  153.         }
  154.  
  155.         /**
  156.          * A CacheManipulator used for testing which does not actually call mlock.
  157.          * This allows many tests to be run even when the operating system does not
  158.          * allow mlock, or only allows limited mlocking.
  159.          */
  160.         @VisibleForTesting
  161.         public static class NoMlockCacheManipulator extends CacheManipulator {
  162.             public void mlock(String identifier, ByteBuffer buffer,
  163.                               long len) throws IOException {
  164.                 LOG.info("mlocking " + identifier);
  165.             }
  166.  
  167.             public long getMemlockLimit() {
  168.                 return 1125899906842624L;
  169.             }
  170.  
  171.             public long getOperatingSystemPageSize() {
  172.                 return 4096;
  173.             }
  174.  
  175.             public boolean verifyCanMlock() {
  176.                 return true;
  177.             }
  178.         }
  179.  
  180.         static {
  181.             if (NativeCodeLoader.isNativeCodeLoaded()) {
  182.                 try {
  183.                     Configuration conf = new Configuration();
  184.                     workaroundNonThreadSafePasswdCalls = conf.getBoolean(
  185.                             WORKAROUND_NON_THREADSAFE_CALLS_KEY,
  186.                             WORKAROUND_NON_THREADSAFE_CALLS_DEFAULT);
  187.  
  188.                     initNative();
  189.                     nativeLoaded = true;
  190.  
  191.                     cacheTimeout = conf.getLong(
  192.                             CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_KEY,
  193.                             CommonConfigurationKeys.HADOOP_SECURITY_UID_NAME_CACHE_TIMEOUT_DEFAULT) *
  194.                             1000;
  195.                     LOG.debug("Initialized cache for IDs to User/Group mapping with a " +
  196.                             " cache timeout of " + cacheTimeout/1000 + " seconds.");
  197.  
  198.                 } catch (Throwable t) {
  199.                     // This can happen if the user has an older version of libhadoop.so
  200.                     // installed - in this case we can continue without native IO
  201.                     // after warning
  202.                     PerformanceAdvisory.LOG.debug("Unable to initialize NativeIO libraries", t);
  203.                 }
  204.             }
  205.         }
  206.  
  207.         /**
  208.          * Return true if the JNI-based native IO extensions are available.
  209.          */
  210.         public static boolean isAvailable() {
  211.             return NativeCodeLoader.isNativeCodeLoaded() && nativeLoaded;
  212.         }
  213.  
  214.         private static void assertCodeLoaded() throws IOException {
  215.             if (!isAvailable()) {
  216.                 throw new IOException("NativeIO was not loaded");
  217.             }
  218.         }
  219.  
  220.         /** Wrapper around open(2) */
  221.         public static native FileDescriptor open(String path, int flags, int mode) throws IOException;
  222.         /** Wrapper around fstat(2) */
  223.         private static native Stat fstat(FileDescriptor fd) throws IOException;
  224.  
  225.         /** Native chmod implementation. On UNIX, it is a wrapper around chmod(2) */
  226.         private static native void chmodImpl(String path, int mode) throws IOException;
  227.  
  228.         public static void chmod(String path, int mode) throws IOException {
  229.             if (!Shell.WINDOWS) {
  230.                 chmodImpl(path, mode);
  231.             } else {
  232.                 try {
  233.                     chmodImpl(path, mode);
  234.                 } catch (NativeIOException nioe) {
  235.                     if (nioe.getErrorCode() == 3) {
  236.                         throw new NativeIOException("No such file or directory",
  237.                                 Errno.ENOENT);
  238.                     } else {
  239.                         LOG.warn(String.format("NativeIO.chmod error (%d): %s",
  240.                                 nioe.getErrorCode(), nioe.getMessage()));
  241.                         throw new NativeIOException("Unknown error", Errno.UNKNOWN);
  242.                     }
  243.                 }
  244.             }
  245.         }
  246.  
  247.         /** Wrapper around posix_fadvise(2) */
  248.         static native void posix_fadvise(
  249.                 FileDescriptor fd, long offset, long len, int flags) throws NativeIOException;
  250.  
  251.         /** Wrapper around sync_file_range(2) */
  252.         static native void sync_file_range(
  253.                 FileDescriptor fd, long offset, long nbytes, int flags) throws NativeIOException;
  254.  
  255.         /**
  256.          * Call posix_fadvise on the given file descriptor. See the manpage
  257.          * for this syscall for more information. On systems where this
  258.          * call is not available, does nothing.
  259.          *
  260.          * @throws NativeIOException if there is an error with the syscall
  261.          */
  262.         static void posixFadviseIfPossible(String identifier,
  263.                                            FileDescriptor fd, long offset, long len, int flags)
  264.                 throws NativeIOException {
  265.             if (nativeLoaded && fadvisePossible) {
  266.                 try {
  267.                     posix_fadvise(fd, offset, len, flags);
  268.                 } catch (UnsupportedOperationException uoe) {
  269.                     fadvisePossible = false;
  270.                 } catch (UnsatisfiedLinkError ule) {
  271.                     fadvisePossible = false;
  272.                 }
  273.             }
  274.         }
  275.  
  276.         /**
  277.          * Call sync_file_range on the given file descriptor. See the manpage
  278.          * for this syscall for more information. On systems where this
  279.          * call is not available, does nothing.
  280.          *
  281.          * @throws NativeIOException if there is an error with the syscall
  282.          */
  283.         public static void syncFileRangeIfPossible(
  284.                 FileDescriptor fd, long offset, long nbytes, int flags)
  285.                 throws NativeIOException {
  286.             if (nativeLoaded && syncFileRangePossible) {
  287.                 try {
  288.                     sync_file_range(fd, offset, nbytes, flags);
  289.                 } catch (UnsupportedOperationException uoe) {
  290.                     syncFileRangePossible = false;
  291.                 } catch (UnsatisfiedLinkError ule) {
  292.                     syncFileRangePossible = false;
  293.                 }
  294.             }
  295.         }
  296.  
  297.         static native void mlock_native(
  298.                 ByteBuffer buffer, long len) throws NativeIOException;
  299.  
  300.         /**
  301.          * Locks the provided direct ByteBuffer into memory, preventing it from
  302.          * swapping out. After a buffer is locked, future accesses will not incur
  303.          * a page fault.
  304.          *
  305.          * See the mlock(2) man page for more information.
  306.          *
  307.          * @throws NativeIOException
  308.          */
  309.         static void mlock(ByteBuffer buffer, long len)
  310.                 throws IOException {
  311.             assertCodeLoaded();
  312.             if (!buffer.isDirect()) {
  313.                 throw new IOException("Cannot mlock a non-direct ByteBuffer");
  314.             }
  315.             mlock_native(buffer, len);
  316.         }
  317.  
  318.         /**
  319.          * Unmaps the block from memory. See munmap(2).
  320.          *
  321.          * There isn't any portable way to unmap a memory region in Java.
  322.          * So we use the sun.nio method here.
  323.          * Note that unmapping a memory region could cause crashes if code
  324.          * continues to reference the unmapped code.  However, if we don't
  325.          * manually unmap the memory, we are dependent on the finalizer to
  326.          * do it, and we have no idea when the finalizer will run.
  327.          *
  328.          * @param buffer    The buffer to unmap.
  329.          */
  330.         public static void munmap(MappedByteBuffer buffer) {
  331.             if (buffer instanceof sun.nio.ch.DirectBuffer) {
  332.                 sun.misc.Cleaner cleaner =
  333.                         ((sun.nio.ch.DirectBuffer)buffer).cleaner();
  334.                 cleaner.clean();
  335.             }
  336.         }
  337.  
  338.         /** Linux only methods used for getOwner() implementation */
  339.         private static native long getUIDforFDOwnerforOwner(FileDescriptor fd) throws IOException;
  340.         private static native String getUserName(long uid) throws IOException;
  341.  
  342.         /**
  343.          * Result type of the fstat call
  344.          */
  345.         public static class Stat {
  346.             private int ownerId, groupId;
  347.             private String owner, group;
  348.             private int mode;
  349.  
  350.             // Mode constants
  351.             public static final int S_IFMT = 0170000;      /* type of file */
  352.             public static final int   S_IFIFO  = 0010000;  /* named pipe (fifo) */
  353.             public static final int   S_IFCHR  = 0020000;  /* character special */
  354.             public static final int   S_IFDIR  = 0040000;  /* directory */
  355.             public static final int   S_IFBLK  = 0060000;  /* block special */
  356.             public static final int   S_IFREG  = 0100000;  /* regular */
  357.             public static final int   S_IFLNK  = 0120000;  /* symbolic link */
  358.             public static final int   S_IFSOCK = 0140000;  /* socket */
  359.             public static final int   S_IFWHT  = 0160000;  /* whiteout */
  360.             public static final int S_ISUID = 0004000;  /* set user id on execution */
  361.             public static final int S_ISGID = 0002000;  /* set group id on execution */
  362.             public static final int S_ISVTX = 0001000;  /* save swapped text even after use */
  363.             public static final int S_IRUSR = 0000400;  /* read permission, owner */
  364.             public static final int S_IWUSR = 0000200;  /* write permission, owner */
  365.             public static final int S_IXUSR = 0000100;  /* execute/search permission, owner */
  366.  
  367.             Stat(int ownerId, int groupId, int mode) {
  368.                 this.ownerId = ownerId;
  369.                 this.groupId = groupId;
  370.                 this.mode = mode;
  371.             }
  372.  
  373.             Stat(String owner, String group, int mode) {
  374.                 if (!Shell.WINDOWS) {
  375.                     this.owner = owner;
  376.                 } else {
  377.                     this.owner = stripDomain(owner);
  378.                 }
  379.                 if (!Shell.WINDOWS) {
  380.                     this.group = group;
  381.                 } else {
  382.                     this.group = stripDomain(group);
  383.                 }
  384.                 this.mode = mode;
  385.             }
  386.  
  387.             @Override
  388.             public String toString() {
  389.                 return "Stat(owner='" + owner + "', group='" + group + "'" +
  390.                         ", mode=" + mode + ")";
  391.             }
  392.  
  393.             public String getOwner() {
  394.                 return owner;
  395.             }
  396.             public String getGroup() {
  397.                 return group;
  398.             }
  399.             public int getMode() {
  400.                 return mode;
  401.             }
  402.         }
  403.  
  404.         /**
  405.          * Returns the file stat for a file descriptor.
  406.          *
  407.          * @param fd file descriptor.
  408.          * @return the file descriptor file stat.
  409.          * @throws IOException thrown if there was an IO error while obtaining the file stat.
  410.          */
  411.         public static Stat getFstat(FileDescriptor fd) throws IOException {
  412.             Stat stat = null;
  413.             if (!Shell.WINDOWS) {
  414.                 stat = fstat(fd);
  415.                 stat.owner = getName(IdCache.USER, stat.ownerId);
  416.                 stat.group = getName(IdCache.GROUP, stat.groupId);
  417.             } else {
  418.                 try {
  419.                     stat = fstat(fd);
  420.                 } catch (NativeIOException nioe) {
  421.                     if (nioe.getErrorCode() == 6) {
  422.                         throw new NativeIOException("The handle is invalid.",
  423.                                 Errno.EBADF);
  424.                     } else {
  425.                         LOG.warn(String.format("NativeIO.getFstat error (%d): %s",
  426.                                 nioe.getErrorCode(), nioe.getMessage()));
  427.                         throw new NativeIOException("Unknown error", Errno.UNKNOWN);
  428.                     }
  429.                 }
  430.             }
  431.             return stat;
  432.         }
  433.  
  434.         private static String getName(IdCache domain, int id) throws IOException {
  435.             Map<Integer, CachedName> idNameCache = (domain == IdCache.USER)
  436.                     ? USER_ID_NAME_CACHE : GROUP_ID_NAME_CACHE;
  437.             String name;
  438.             CachedName cachedName = idNameCache.get(id);
  439.             long now = System.currentTimeMillis();
  440.             if (cachedName != null && (cachedName.timestamp + cacheTimeout) > now) {
  441.                 name = cachedName.name;
  442.             } else {
  443.                 name = (domain == IdCache.USER) ? getUserName(id) : getGroupName(id);
  444.                 if (LOG.isDebugEnabled()) {
  445.                     String type = (domain == IdCache.USER) ? "UserName" : "GroupName";
  446.                     LOG.debug("Got " + type + " " + name + " for ID " + id +
  447.                             " from the native implementation");
  448.                 }
  449.                 cachedName = new CachedName(name, now);
  450.                 idNameCache.put(id, cachedName);
  451.             }
  452.             return name;
  453.         }
  454.  
  455.         static native String getUserName(int uid) throws IOException;
  456.         static native String getGroupName(int uid) throws IOException;
  457.  
  458.         private static class CachedName {
  459.             final long timestamp;
  460.             final String name;
  461.  
  462.             public CachedName(String name, long timestamp) {
  463.                 this.name = name;
  464.                 this.timestamp = timestamp;
  465.             }
  466.         }
  467.  
  468.         private static final Map<Integer, CachedName> USER_ID_NAME_CACHE =
  469.                 new ConcurrentHashMap<Integer, CachedName>();
  470.  
  471.         private static final Map<Integer, CachedName> GROUP_ID_NAME_CACHE =
  472.                 new ConcurrentHashMap<Integer, CachedName>();
  473.  
  474.         private enum IdCache { USER, GROUP }
  475.  
  476.         public final static int MMAP_PROT_READ = 0x1;
  477.         public final static int MMAP_PROT_WRITE = 0x2;
  478.         public final static int MMAP_PROT_EXEC = 0x4;
  479.  
  480.         public static native long mmap(FileDescriptor fd, int prot,
  481.                                        boolean shared, long length) throws IOException;
  482.  
  483.         public static native void munmap(long addr, long length)
  484.                 throws IOException;
  485.     }
  486.  
  487.     private static boolean workaroundNonThreadSafePasswdCalls = false;
  488.  
  489.  
  490.     public static class Windows {
  491.         // Flags for CreateFile() call on Windows
  492.         public static final long GENERIC_READ = 0x80000000L;
  493.         public static final long GENERIC_WRITE = 0x40000000L;
  494.  
  495.         public static final long FILE_SHARE_READ = 0x00000001L;
  496.         public static final long FILE_SHARE_WRITE = 0x00000002L;
  497.         public static final long FILE_SHARE_DELETE = 0x00000004L;
  498.  
  499.         public static final long CREATE_NEW = 1;
  500.         public static final long CREATE_ALWAYS = 2;
  501.         public static final long OPEN_EXISTING = 3;
  502.         public static final long OPEN_ALWAYS = 4;
  503.         public static final long TRUNCATE_EXISTING = 5;
  504.  
  505.         public static final long FILE_BEGIN = 0;
  506.         public static final long FILE_CURRENT = 1;
  507.         public static final long FILE_END = 2;
  508.  
  509.         public static final long FILE_ATTRIBUTE_NORMAL = 0x00000080L;
  510.  
  511.         /**
  512.          * Create a directory with permissions set to the specified mode.  By setting
  513.          * permissions at creation time, we avoid issues related to the user lacking
  514.          * WRITE_DAC rights on subsequent chmod calls.  One example where this can
  515.          * occur is writing to an SMB share where the user does not have Full Control
  516.          * rights, and therefore WRITE_DAC is denied.
  517.          *
  518.          * @param path directory to create
  519.          * @param mode permissions of new directory
  520.          * @throws IOException if there is an I/O error
  521.          */
  522.         public static void createDirectoryWithMode(File path, int mode)
  523.                 throws IOException {
  524.             createDirectoryWithMode0(path.getAbsolutePath(), mode);
  525.         }
  526.  
  527.         /** Wrapper around CreateDirectory() on Windows */
  528.         private static native void createDirectoryWithMode0(String path, int mode)
  529.                 throws NativeIOException;
  530.  
  531.         /** Wrapper around CreateFile() on Windows */
  532.         public static native FileDescriptor createFile(String path,
  533.                                                        long desiredAccess, long shareMode, long creationDisposition)
  534.                 throws IOException;
  535.  
  536.         /**
  537.          * Create a file for write with permissions set to the specified mode.  By
  538.          * setting permissions at creation time, we avoid issues related to the user
  539.          * lacking WRITE_DAC rights on subsequent chmod calls.  One example where
  540.          * this can occur is writing to an SMB share where the user does not have
  541.          * Full Control rights, and therefore WRITE_DAC is denied.
  542.          *
  543.          * This method mimics the semantics implemented by the JDK in
  544.          * {@link FileOutputStream}.  The file is opened for truncate or
  545.          * append, the sharing mode allows other readers and writers, and paths
  546.          * longer than MAX_PATH are supported.  (See io_util_md.c in the JDK.)
  547.          *
  548.          * @param path file to create
  549.          * @param append if true, then open file for append
  550.          * @param mode permissions of new directory
  551.          * @return FileOutputStream of opened file
  552.          * @throws IOException if there is an I/O error
  553.          */
  554.         public static FileOutputStream createFileOutputStreamWithMode(File path,
  555.                                                                       boolean append, int mode) throws IOException {
  556.             long desiredAccess = GENERIC_WRITE;
  557.             long shareMode = FILE_SHARE_READ | FILE_SHARE_WRITE;
  558.             long creationDisposition = append ? OPEN_ALWAYS : CREATE_ALWAYS;
  559.             return new FileOutputStream(createFileWithMode0(path.getAbsolutePath(),
  560.                     desiredAccess, shareMode, creationDisposition, mode));
  561.         }
  562.  
  563.         /** Wrapper around CreateFile() with security descriptor on Windows */
  564.         private static native FileDescriptor createFileWithMode0(String path,
  565.                                                                  long desiredAccess, long shareMode, long creationDisposition, int mode)
  566.                 throws NativeIOException;
  567.  
  568.         /** Wrapper around SetFilePointer() on Windows */
  569.         public static native long setFilePointer(FileDescriptor fd,
  570.                                                  long distanceToMove, long moveMethod) throws IOException;
  571.  
  572.         /** Windows only methods used for getOwner() implementation */
  573.         private static native String getOwner(FileDescriptor fd) throws IOException;
  574.  
  575.         /** Supported list of Windows access right flags */
  576.         public static enum AccessRight {
  577.             ACCESS_READ (0x0001),      // FILE_READ_DATA
  578.             ACCESS_WRITE (0x0002),     // FILE_WRITE_DATA
  579.             ACCESS_EXECUTE (0x0020);   // FILE_EXECUTE
  580.  
  581.             private final int accessRight;
  582.             AccessRight(int access) {
  583.                 accessRight = access;
  584.             }
  585.  
  586.             public int accessRight() {
  587.                 return accessRight;
  588.             }
  589.         };
  590.  
  591.         /** Windows only method used to check if the current process has requested
  592.          *  access rights on the given path. */
  593.         private static native boolean access0(String path, int requestedAccess);
  594.  
  595.         /**
  596.          * Checks whether the current process has desired access rights on
  597.          * the given path.
  598.          *
  599.          * Longer term this native function can be substituted with JDK7
  600.          * function Files#isReadable, isWritable, isExecutable.
  601.          *
  602.          * @param path input path
  603.          * @param desiredAccess ACCESS_READ, ACCESS_WRITE or ACCESS_EXECUTE
  604.          * @return true if access is allowed
  605.          * @throws IOException I/O exception on error
  606.          */
  607.         public static boolean access(String path, AccessRight desiredAccess)
  608.                 throws IOException {
  609.             //return access0(path, desiredAccess.accessRight());
  610.             return true;
  611.         }
  612.  
  613.         /**
  614.          * Extends both the minimum and maximum working set size of the current
  615.          * process.  This method gets the current minimum and maximum working set
  616.          * size, adds the requested amount to each and then sets the minimum and
  617.          * maximum working set size to the new values.  Controlling the working set
  618.          * size of the process also controls the amount of memory it can lock.
  619.          *
  620.          * @param delta amount to increment minimum and maximum working set size
  621.          * @throws IOException for any error
  622.          * @see POSIX#mlock(ByteBuffer, long)
  623.          */
  624.         public static native void extendWorkingSetSize(long delta) throws IOException;
  625.  
  626.         static {
  627.             if (NativeCodeLoader.isNativeCodeLoaded()) {
  628.                 try {
  629.                     initNative();
  630.                     nativeLoaded = true;
  631.                 } catch (Throwable t) {
  632.                     // This can happen if the user has an older version of libhadoop.so
  633.                     // installed - in this case we can continue without native IO
  634.                     // after warning
  635.                     PerformanceAdvisory.LOG.debug("Unable to initialize NativeIO libraries", t);
  636.                 }
  637.             }
  638.         }
  639.     }
  640.  
  641.     private static final Log LOG = LogFactory.getLog(NativeIO.class);
  642.  
  643.     private static boolean nativeLoaded = false;
  644.  
  645.     static {
  646.         if (NativeCodeLoader.isNativeCodeLoaded()) {
  647.             try {
  648.                 initNative();
  649.                 nativeLoaded = true;
  650.             } catch (Throwable t) {
  651.                 // This can happen if the user has an older version of libhadoop.so
  652.                 // installed - in this case we can continue without native IO
  653.                 // after warning
  654.                 PerformanceAdvisory.LOG.debug("Unable to initialize NativeIO libraries", t);
  655.             }
  656.         }
  657.     }
  658.  
  659.     /**
  660.      * Return true if the JNI-based native IO extensions are available.
  661.      */
  662.     public static boolean isAvailable() {
  663.         return NativeCodeLoader.isNativeCodeLoaded() && nativeLoaded;
  664.     }
  665.  
  666.     /** Initialize the JNI method ID and class ID cache */
  667.     private static native void initNative();
  668.  
  669.     /**
  670.      * Get the maximum number of bytes that can be locked into memory at any
  671.      * given point.
  672.      *
  673.      * @return 0 if no bytes can be locked into memory;
  674.      *         Long.MAX_VALUE if there is no limit;
  675.      *         The number of bytes that can be locked into memory otherwise.
  676.      */
  677.     static long getMemlockLimit() {
  678.         return isAvailable() ? getMemlockLimit0() : 0;
  679.     }
  680.  
  681.     private static native long getMemlockLimit0();
  682.  
  683.     /**
  684.      * @return the operating system's page size.
  685.      */
  686.     static long getOperatingSystemPageSize() {
  687.         try {
  688.             Field f = Unsafe.class.getDeclaredField("theUnsafe");
  689.             f.setAccessible(true);
  690.             Unsafe unsafe = (Unsafe)f.get(null);
  691.             return unsafe.pageSize();
  692.         } catch (Throwable e) {
  693.             LOG.warn("Unable to get operating system page size.  Guessing 4096.", e);
  694.             return 4096;
  695.         }
  696.     }
  697.  
  698.     private static class CachedUid {
  699.         final long timestamp;
  700.         final String username;
  701.         public CachedUid(String username, long timestamp) {
  702.             this.timestamp = timestamp;
  703.             this.username = username;
  704.         }
  705.     }
  706.     private static final Map<Long, CachedUid> uidCache =
  707.             new ConcurrentHashMap<Long, CachedUid>();
  708.     private static long cacheTimeout;
  709.     private static boolean initialized = false;
  710.  
  711.     /**
  712.      * The Windows logon name has two part, NetBIOS domain name and
  713.      * user account name, of the format DOMAIN\UserName. This method
  714.      * will remove the domain part of the full logon name.
  715.      *
  716.      * @param Fthe full principal name containing the domain
  717.      * @return name with domain removed
  718.      */
  719.     private static String stripDomain(String name) {
  720.         int i = name.indexOf('\\');
  721.         if (i != -1)
  722.             name = name.substring(i + 1);
  723.         return name;
  724.     }
  725.  
  726.     public static String getOwner(FileDescriptor fd) throws IOException {
  727.         ensureInitialized();
  728.         if (Shell.WINDOWS) {
  729.             String owner = Windows.getOwner(fd);
  730.             owner = stripDomain(owner);
  731.             return owner;
  732.         } else {
  733.             long uid = POSIX.getUIDforFDOwnerforOwner(fd);
  734.             CachedUid cUid = uidCache.get(uid);
  735.             long now = System.currentTimeMillis();
  736.             if (cUid != null && (cUid.timestamp + cacheTimeout) > now) {
  737.                 return cUid.username;
  738.             }
  739.             String user = POSIX.getUserName(uid);
  740.             LOG.info("Got UserName " + user + " for UID " + uid
  741.                     + " from the native implementation");
  742.             cUid = new CachedUid(user, now);
  743.             uidCache.put(uid, cUid);
  744.             return user;
  745.         }
  746.     }
  747.  
  748.     /**
  749.      * Create a FileInputStream that shares delete permission on the
  750.      * file opened, i.e. other process can delete the file the
  751.      * FileInputStream is reading. Only Windows implementation uses
  752.      * the native interface.
  753.      */
  754.     public static FileInputStream getShareDeleteFileInputStream(File f)
  755.             throws IOException {
  756.         if (!Shell.WINDOWS) {
  757.             // On Linux the default FileInputStream shares delete permission
  758.             // on the file opened.
  759.             //
  760.             return new FileInputStream(f);
  761.         } else {
  762.             // Use Windows native interface to create a FileInputStream that
  763.             // shares delete permission on the file opened.
  764.             //
  765.             FileDescriptor fd = Windows.createFile(
  766.                     f.getAbsolutePath(),
  767.                     Windows.GENERIC_READ,
  768.                     Windows.FILE_SHARE_READ |
  769.                             Windows.FILE_SHARE_WRITE |
  770.                             Windows.FILE_SHARE_DELETE,
  771.                     Windows.OPEN_EXISTING);
  772.             return new FileInputStream(fd);
  773.         }
  774.     }
  775.  
  776.     /**
  777.      * Create a FileInputStream that shares delete permission on the
  778.      * file opened at a given offset, i.e. other process can delete
  779.      * the file the FileInputStream is reading. Only Windows implementation
  780.      * uses the native interface.
  781.      */
  782.     public static FileInputStream getShareDeleteFileInputStream(File f, long seekOffset)
  783.             throws IOException {
  784.         if (!Shell.WINDOWS) {
  785.             RandomAccessFile rf = new RandomAccessFile(f, "r");
  786.             if (seekOffset > 0) {
  787.                 rf.seek(seekOffset);
  788.             }
  789.             return new FileInputStream(rf.getFD());
  790.         } else {
  791.             // Use Windows native interface to create a FileInputStream that
  792.             // shares delete permission on the file opened, and set it to the
  793.             // given offset.
  794.             //
  795.             FileDescriptor fd = Windows.createFile(
  796.                     f.getAbsolutePath(),
  797.                     Windows.GENERIC_READ,
  798.                     Windows.FILE_SHARE_READ |
  799.                             Windows.FILE_SHARE_WRITE |
  800.                             Windows.FILE_SHARE_DELETE,
  801.                     Windows.OPEN_EXISTING);
  802.             if (seekOffset > 0)
  803.                 Windows.setFilePointer(fd, seekOffset, Windows.FILE_BEGIN);
  804.             return new FileInputStream(fd);
  805.         }
  806.     }
  807.  
  808.     /**
  809.      * Create the specified File for write access, ensuring that it does not exist.
  810.      * @param f the file that we want to create
  811.      * @param permissions we want to have on the file (if security is enabled)
  812.      *
  813.      * @throws AlreadyExistsException if the file already exists
  814.      * @throws IOException if any other error occurred
  815.      */
  816.     public static FileOutputStream getCreateForWriteFileOutputStream(File f, int permissions)
  817.             throws IOException {
  818.         if (!Shell.WINDOWS) {
  819.             // Use the native wrapper around open(2)
  820.             try {
  821.                 FileDescriptor fd = POSIX.open(f.getAbsolutePath(),
  822.                         POSIX.O_WRONLY | POSIX.O_CREAT
  823.                                 | POSIX.O_EXCL, permissions);
  824.                 return new FileOutputStream(fd);
  825.             } catch (NativeIOException nioe) {
  826.                 if (nioe.getErrno() == Errno.EEXIST) {
  827.                     throw new AlreadyExistsException(nioe);
  828.                 }
  829.                 throw nioe;
  830.             }
  831.         } else {
  832.             // Use the Windows native APIs to create equivalent FileOutputStream
  833.             try {
  834.                 FileDescriptor fd = Windows.createFile(f.getCanonicalPath(),
  835.                         Windows.GENERIC_WRITE,
  836.                         Windows.FILE_SHARE_DELETE
  837.                                 | Windows.FILE_SHARE_READ
  838.                                 | Windows.FILE_SHARE_WRITE,
  839.                         Windows.CREATE_NEW);
  840.                 POSIX.chmod(f.getCanonicalPath(), permissions);
  841.                 return new FileOutputStream(fd);
  842.             } catch (NativeIOException nioe) {
  843.                 if (nioe.getErrorCode() == 80) {
  844.                     // ERROR_FILE_EXISTS
  845.                     // 80 (0x50)
  846.                     // The file exists
  847.                     throw new AlreadyExistsException(nioe);
  848.                 }
  849.                 throw nioe;
  850.             }
  851.         }
  852.     }
  853.  
  854.     private synchronized static void ensureInitialized() {
  855.         if (!initialized) {
  856.             cacheTimeout =
  857.                     new Configuration().getLong("hadoop.security.uid.cache.secs",
  858.                             4*60*60) * 1000;
  859.             LOG.info("Initialized cache for UID to User mapping with a cache" +
  860.                     " timeout of " + cacheTimeout/1000 + " seconds.");
  861.             initialized = true;
  862.         }
  863.     }
  864.  
  865.     /**
  866.      * A version of renameTo that throws a descriptive exception when it fails.
  867.      *
  868.      * @param src                  The source path
  869.      * @param dst                  The destination path
  870.      *
  871.      * @throws NativeIOException   On failure.
  872.      */
  873.     public static void renameTo(File src, File dst)
  874.             throws IOException {
  875.         if (!nativeLoaded) {
  876.             if (!src.renameTo(dst)) {
  877.                 throw new IOException("renameTo(src=" + src + ", dst=" +
  878.                         dst + ") failed.");
  879.             }
  880.         } else {
  881.             renameTo0(src.getAbsolutePath(), dst.getAbsolutePath());
  882.         }
  883.     }
  884.  
  885.     public static void link(File src, File dst) throws IOException {
  886.         if (!nativeLoaded) {
  887.             HardLink.createHardLink(src, dst);
  888.         } else {
  889.             link0(src.getAbsolutePath(), dst.getAbsolutePath());
  890.         }
  891.     }
  892.  
  893.     /**
  894.      * A version of renameTo that throws a descriptive exception when it fails.
  895.      *
  896.      * @param src                  The source path
  897.      * @param dst                  The destination path
  898.      *
  899.      * @throws NativeIOException   On failure.
  900.      */
  901.     private static native void renameTo0(String src, String dst)
  902.             throws NativeIOException;
  903.  
  904.     private static native void link0(String src, String dst)
  905.             throws NativeIOException;
  906.  
  907.     /**
  908.      * Unbuffered file copy from src to dst without tainting OS buffer cache
  909.      *
  910.      * In POSIX platform:
  911.      * It uses FileChannel#transferTo() which internally attempts
  912.      * unbuffered IO on OS with native sendfile64() support and falls back to
  913.      * buffered IO otherwise.
  914.      *
  915.      * It minimizes the number of FileChannel#transferTo call by passing the the
  916.      * src file size directly instead of a smaller size as the 3rd parameter.
  917.      * This saves the number of sendfile64() system call when native sendfile64()
  918.      * is supported. In the two fall back cases where sendfile is not supported,
  919.      * FileChannle#transferTo already has its own batching of size 8 MB and 8 KB,
  920.      * respectively.
  921.      *
  922.      * In Windows Platform:
  923.      * It uses its own native wrapper of CopyFileEx with COPY_FILE_NO_BUFFERING
  924.      * flag, which is supported on Windows Server 2008 and above.
  925.      *
  926.      * Ideally, we should use FileChannel#transferTo() across both POSIX and Windows
  927.      * platform. Unfortunately, the wrapper(Java_sun_nio_ch_FileChannelImpl_transferTo0)
  928.      * used by FileChannel#transferTo for unbuffered IO is not implemented on Windows.
  929.      * Based on OpenJDK 6/7/8 source code, Java_sun_nio_ch_FileChannelImpl_transferTo0
  930.      * on Windows simply returns IOS_UNSUPPORTED.
  931.      *
  932.      * Note: This simple native wrapper does minimal parameter checking before copy and
  933.      * consistency check (e.g., size) after copy.
  934.      * It is recommended to use wrapper function like
  935.      * the Storage#nativeCopyFileUnbuffered() function in hadoop-hdfs with pre/post copy
  936.      * checks.
  937.      *
  938.      * @param src                  The source path
  939.      * @param dst                  The destination path
  940.      * @throws IOException
  941.      */
  942.     public static void copyFileUnbuffered(File src, File dst) throws IOException {
  943.         if (nativeLoaded && Shell.WINDOWS) {
  944.             copyFileUnbuffered0(src.getAbsolutePath(), dst.getAbsolutePath());
  945.         } else {
  946.             FileInputStream fis = null;
  947.             FileOutputStream fos = null;
  948.             FileChannel input = null;
  949.             FileChannel output = null;
  950.             try {
  951.                 fis = new FileInputStream(src);
  952.                 fos = new FileOutputStream(dst);
  953.                 input = fis.getChannel();
  954.                 output = fos.getChannel();
  955.                 long remaining = input.size();
  956.                 long position = 0;
  957.                 long transferred = 0;
  958.                 while (remaining > 0) {
  959.                     transferred = input.transferTo(position, remaining, output);
  960.                     remaining -= transferred;
  961.                     position += transferred;
  962.                 }
  963.             } finally {
  964.                 IOUtils.cleanup(LOG, output);
  965.                 IOUtils.cleanup(LOG, fos);
  966.                 IOUtils.cleanup(LOG, input);
  967.                 IOUtils.cleanup(LOG, fis);
  968.             }
  969.         }
  970.     }
  971.  
  972.     private static native void copyFileUnbuffered0(String src, String dst)
  973.             throws NativeIOException;
  974. }

　第二种：在window-->System32文件夹中添加一个插件

 

 下载地址：https://github.com/amihalik/hadoop-common-2.6.0-bin，此文件需与hadoop版本一致，不能太低，否则无效

 

亲身试验，可用

 

此博客参考：https://blog.csdn.net/weixin_42229056/article/details/82686172

原文链接：http://www.cnblogs.com/zhuzhu-you/p/14037949.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

MapReduce热门文章

  • mapreduce运行问题 ：java.io.EOFException
  • 云计算实验：Java?MapReduce编程
  • MapReduce案例-好友推荐
  • 【Hadoop】：MapReduce实现序列化
  • 03_MapReduce框架原理_3.4 InputSplit 切片类(源码)
  • 【Hadoop】：MapReduce核心思想
  • Java实现MapReduce Wordcount案例
  • MapReduce与Yarn 的详细工作流程分析
  • 【概念】详解MapReduce原理
  • MapReduce报错Exception in thread &quot;main&quot; java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio

    • MapReduce推荐文章

  • 03_MapReduce框架原理_3.4 InputSplit 切片类(源码)
  • 云计算实验：Java?MapReduce编程
  • 【Hadoop】：MapReduce实现序列化
  • MapReduce报错Exception in thread &quot;main&quot; java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio
  • Java实现MapReduce Wordcount案例

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号