HDF5 plugin to support defining hard links in XML layout

[ulrikpedersen]

In order for the HDF5 plugin to be able to write truly NeXus compatible files, we have identified the need for an additional feature which allow defining HDF5 hard links to specific datasets in the XML layout definition.

• The XSD schema will need to be extended with a "hardlink" element.
• The XML parsing code need to read this new "hardlink" element and add it to the internal structure of HdfElement (i.e. a new HdfElement subclass is required)
• The creation of the HDF5 file with all the elements need to be extended to support creating the hard links.

Reference comments in #28 and #29

Pull request #30 has added a comment where a 'default' hard-link to the main dataset need to go.

[prjemian]

Here is an example from an area detector 1.x template file that I have used in the past to indicate to my post-processing step how to construct the link:

    <link_rules type="UserGroup">
        <link>
            <Attr name="source" outtype="NX_CHAR" type="CONST">/entry/data</Attr>
            <Attr name="target" outtype="NX_CHAR" type="CONST">/entry/instrument/detector/data</Attr>
            <!-- make data appear under detector, as well -->
       </link>
    </link_rules>

The link_rules element is specified only once in a template file and provides a list of links to be constructed after all the other HDF5 data and groups are created. Each link has two attributes as shown. To match the interface of the H5Lcreate_hard() function,

herr_t H5Lcreate_hard( hid_t obj_loc_id, const char *obj_name, hid_t link_loc_id, const char *link_name, hid_t lcpl_id, hid_t lapl_id ) 

it might be easier to rewrite:

    <link_rules type="UserGroup">
        <link>
            <Attr name="source_group" outtype="NX_CHAR" type="CONST">/entry</Attr>
            <Attr name="source_name" outtype="NX_CHAR" type="CONST">data</Attr>
            <Attr name="target_group" outtype="NX_CHAR" type="CONST">/entry/instrument/detector</Attr>
            <Attr name="target_name" outtype="NX_CHAR" type="CONST">data</Attr>
            <!-- make data appear under detector, as well -->
       </link>
    </link_rules>

see: http://www.hdfgroup.org/HDF5/doc/RM/RM_H5L.html#Link-CreateHard

[prjemian]

python code to create a specific hard link in a file named h5File

import h5py

def create_missing_link(h5File):
    h5 = h5py.File(h5File, "a")
    parent = h5['/entry/data']
    ds = parent['data']
    makeLink(parent, ds, '/entry/instrument/detector/data')
    h5.close()

def makeLink(parent, sourceObject, targetName):
    '''
    create an internal NeXus (hard) link in an HDF5 file
    '''
    if not 'target' in sourceObject.attrs:
        # NeXus link, NOT an HDF5 link!
        sourceObject.attrs["target"] = str(sourceObject.name)
    import h5py.h5g
    parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

[MarkRivers]

Folks,

I am starting to work on implementing links into the HDF5 file plugin.

I have a question about what level links should appear in the XML file. Here is what I am asking based on the hdf5_xml_layout_schema.xsd file.

I defined a hardlink element type.

<xs:complexType name="hardlinkType">
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="source" type="xs:string" use="required" />
/xs:complexType

I can think of 2 ways to do this. A hardlink can be under a “group”.

<xs:complexType name="groupType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="dataset" type="datasetType" />
<xs:element name="attribute" type="attributeType" />
<xs:element name="hardlink" type="hardlinkType" />
<xs:element name="group" type="groupType" />
/xs:choice
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="ndattr_default" type="xs:boolean" use="optional" default="false" />
/xs:complexType

This would be used as follows in an XML file:

Or a hardlink could be a top-level element:

<xs:element name="xml">
xs:complexType
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="global" type="globalType" />
<xs:element name="group" type="groupType" />
<xs:element name="hardlink" type="hardlinkType" />
/xs:choice
/xs:complexType
/xs:element

This would be used as follows:

<group name="data">
  <attribute name="NX_class" source="constant" value="NXdata" type="string"></attribute>
</group>              <!-- end group data -->

<!-- needs a hard link from /entry/instrument/detector/data to /entry/data/data -->

I think the first syntax is easier to use, but it forces hardlinks to belong to groups. Is this OK?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Monday, September 22, 2014 1:40 PM
To: areaDetector/ADCore
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML layout (#31)

python code to create a specific hard link in a file named h5File

import h5py

def create_missing_link(h5File):

h5 = h5py.File(h5File, "a")

parent = h5['/entry/data']

ds = parent['data']

makeLink(parent, ds, '/entry/instrument/detector/data')

h5.close()

def makeLink(parent, sourceObject, targetName):

'''

create an internal NeXus (hard) link in an HDF5 file

'''

if not 'target' in sourceObject.attrs:

    # NeXus link, NOT an HDF5 link!

    sourceObject.attrs["target"] = str(sourceObject.name)

import h5py.h5g

parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-56419326.

[prjemian]

I like the first syntax (links defined under the group), it will look
more obvious.

On 10/6/2014 1:00 PM, Mark Rivers wrote:

Folks,

I am starting to work on implementing links into the HDF5 file plugin.

I have a question about what level links should appear in the XML file.
Here is what I am asking based on the hdf5_xml_layout_schema.xsd file.

I defined a hardlink element type.

<xs:complexType name="hardlinkType">
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="source" type="xs:string" use="required" />
/xs:complexType

I can think of 2 ways to do this. A hardlink can be under a “group”.

<xs:complexType name="groupType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="dataset" type="datasetType" />
<xs:element name="attribute" type="attributeType" />
<xs:element name="hardlink" type="hardlinkType" />
<xs:element name="group" type="groupType" />
/xs:choice
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="ndattr_default" type="xs:boolean" use="optional"
default="false" />
/xs:complexType

This would be used as follows in an XML file:

Or a hardlink could be a top-level element:

<xs:element name="xml">
xs:complexType
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="global" type="globalType" />
<xs:element name="group" type="groupType" />
<xs:element name="hardlink" type="hardlinkType" />
/xs:choice
/xs:complexType
/xs:element

This would be used as follows:

I think the first syntax is easier to use, but it forces hardlinks to
belong to groups. Is this OK?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Monday, September 22, 2014 1:40 PM
To: areaDetector/ADCore
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML
layout (#31)

python code to create a specific hard link in a file named h5File

import h5py

def create_missing_link(h5File):

h5 = h5py.File(h5File, "a")

parent = h5['/entry/data']

ds = parent['data']

makeLink(parent, ds, '/entry/instrument/detector/data')

h5.close()

def makeLink(parent, sourceObject, targetName):

'''

create an internal NeXus (hard) link in an HDF5 file

'''

if not 'target' in sourceObject.attrs:

NeXus link, NOT an HDF5 link!

sourceObject.attrs["target"] = str(sourceObject.name)

import h5py.h5g

parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/31#issuecomment-56419326.

—
Reply to this email directly or view it on GitHub
#31 (comment).

[ulrikpedersen]

Hi Mark,
Thanks for looking into this (sorry I just noticed your comment on this entry after I sent my email...)

OK, so I agree with Pete: the first syntax is best. In fact I think this is the only correct way. All HDF elements (groups, datasets, hardlinks) must belong to groups - even if the group is the root ('/') of the tree. With your first option, you are enforcing this - whereas the second option leaves some ambiguity.

So just to confirm; the hardlinkType will look like this:

<!-- The hardlink element -->
<xs:complexType name="hardlinkType">
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="source" type="xs:string" use="required" />
</xs:complexType>

And it will be added as an optional entry to the groupType, at the same level as datasets, attributes and other groups:

<!-- The group element can contain other elements: datasets, attributes, hardlinks, and other groups -->
<xs:complexType name="groupType">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="dataset" type="datasetType" />
        <xs:element name="attribute" type="attributeType" />
        <xs:element name="hardlink" type="hardlinkType" />
        <xs:element name="group" type="groupType" />
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="ndattr_default" type="xs:boolean" use="optional" default="false" />
</xs:complexType> 

Finally, it can be used like this example (where a dataset has been defined in the /entry/instrument/detector/data):

<group name="data">
    <attribute name="NX_class" source="constant" value="NXdata" type="string"></attribute>
    <!-- needs a hard link from /entry/instrument/detector/data to /entry/data/data -->
    <hardlink name="data" source="/entry/instrument/detector/data"></hardlink>
</group> <!-- end group data -->

Cheers,
Ulrik

[MarkRivers]

Folks,

I now have this working. I've changed the default layout to include the hardlink.

This is an h5dump of a resulting file:

corvette:~/scratch>h5dump --header hdf5_no_xml_file_012.h5
HDF5 "hdf5_no_xml_file_012.h5" {
GROUP "/" {
GROUP "entry" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 7;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
GROUP "data" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
DATASET "data" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 1024, 1024 ) / ( 1024, 1024 ) }
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 3;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "signal" {
DATATYPE H5T_STD_I32LE
DATASPACE SCALAR
}
}
}
GROUP "instrument" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 12;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
GROUP "NDAttributes" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 12;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "hostname" {
DATATYPE H5T_STRING {
STRSIZE 25;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
GROUP "detector" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 10;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
GROUP "NDAttributes" {
ATTRIBUTE "NX_class" {
DATATYPE H5T_STRING {
STRSIZE 12;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
DATASET "ColorMode" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 10 ) / ( 10 ) }
ATTRIBUTE "description" {
DATATYPE H5T_STRING {
STRSIZE 10;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
ATTRIBUTE "source" {
DATATYPE H5T_STRING {
STRSIZE 6;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
}
DATASPACE SCALAR
}
}
}
DATASET "data" {
HARDLINK "/entry/data/data"
}
}
GROUP "performance" {
DATASET "timestamp" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 1, 5 ) / ( 1, 5 ) }
}
}
}
}
}
}

Note that it appears the the data is in /entry/data/data and that /entry/instrument/detector/data is the hardlink. This is OK, both of those paths are really hardlinks to the location in the file, and it needs to decide which one to display as a link and which to display as the "data".

I've committed a new branch "rivers_hdf5" on github. Please try to test it and let me know of any problems.

Mark

---

From: Ulrik Kofoed Pedersen [notifications@github.com]
Sent: Tuesday, October 07, 2014 4:05 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML layout (#31)

Hi Mark,
Thanks for looking into this (sorry I just noticed your comment on this entry after I sent my email...)

OK, so I agree with Pete: the first syntax is best. In fact I think this is the only correct way. All HDF elements (groups, datasets, hardlinks) must belong to groups - even if the group is the root ('/') of the tree. With your first option, you are enforcing this - whereas the second option leaves some ambiguity.

So just to confirm; the hardlinkType will look like this:

<xs:complexType name="hardlinkType">
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="source" type="xs:string" use="required" />
/xs:complexType

And it will be added as an optional entry to the groupType, at the same level as datasets, attributes and other groups:

<xs:complexType name="groupType">
<xs:choice minOccurs="0" maxOccurs="unbounded">
<xs:element name="dataset" type="datasetType" />
<xs:element name="attribute" type="attributeType" />
<xs:element name="hardlink" type="hardlinkType" />
<xs:element name="group" type="groupType" />
/xs:choice
<xs:attribute name="name" type="xs:string" use="required" />
<xs:attribute name="ndattr_default" type="xs:boolean" use="optional" default="false" />
/xs:complexType

Finally, it can be used like this example (where a dataset has been defined in the /entry/instrument/detector/data):

Cheers,
Ulrik

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-58156271.

[ulrikpedersen]

Thanks Mark,

I just ran a basic test of our use-case:

• Stream mode, 10 images
• test script here: adclientxmlhdf.py
• XML file with new hardlink defined: layout.xml

The result is a hardlinked dataset with all the attributes of the main dataset. I am a little surprised to see that the hardlink appears as the main dataset - but I guess that is just down to it being an actual hard link; i.e. indistinguishable from the real data to a client tool. I might ask around about that behavior.

Here is a h5ls (notice /entry/data/data and /entry/detector/data1 - the latter is the main dataset):

[up45@pc0009 testHdfXml]$ h5ls -r out.h5 
/                        Group
/entry                   Group
/entry/attributes        Group
/entry/data              Group
/entry/data/data         Dataset {10/Inf, 1024, 1024}
/entry/detector          Group
/entry/detector/data1    Dataset, same as /entry/data/data
/entry/detector/data2    Dataset {1/Inf, 1024, 1024}
/entry/detector/data3    Dataset {1/Inf, 1024, 1024}
/entry/instruments       Group
/entry/instruments/ColorMode Dataset {10}
/entry/instruments/timestamp Dataset {10, 5}

Cheers,
Ulrik

[ulrikpedersen]

Just a quick comment to let you know that I have discussed this funny behaviour of a hardlinks with my colleague Tobias Richter who has a lot of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the main dataset when viewed with h5ls or h5dump. Tobias confirmed that this is indeed a behaviour that he has noticed when working with NeXus/HDF5 files.

We think it maybe due to datasets and hardlinks sharing the same objectID (like inodes on a filesystem). The applications h5dump and h5ls will iterate through the files objectIDs and will probably select the first one it finds as the 'main' dataset - with all others of the same objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood are just references to file system locations - apparently even if you delete the main dataset, but still have a hardlink, pointing to it, the real data is not deleted (or lost) from disk.

[MarkRivers]

I noticed this same behavior yesterday, and asked Pete Jemian about it. He basically said the same thing as Tobias.

This is his reply (and my message):

---

I confirm your experience with the difference between

plotting /entry/data/data

and getting an error trying to plot /entry/instrument/detector/data

The nature of hard links in HDF5 is similar to hard links in a file

system. Any entry in the HDF5 directory (such as

/entry/instrument/detector/data) is a hard link to the data object

stored in the HDF5 file. When you call H5Lcreate(), you are making an

additional link from another place in the directory to the same data

object. For convenience, the H5Lcreate() allows you to point at an

existing directory item rather than the actual data reference.

If you examine the same file with HdfView, both directory items

/entry/data/data and /entry/instrument/detector/data have the same

(5732,2) data reference. Neither one of these items is the "original"

data, from the point of view of HDF5 storage. They are both references

to the same data object. If you delete one of them, the data object

remains. Once you delete all items that reference a data object, then

HDF5 can delete the data object from the file.

h5dump cannot tell which directory item was the one created when the

data object was created. It picks one of them.

NeXus adds an additional attribute, "target", to the first directory

item to indicate which can be considered the "original". Aside from

guiding us readers of these files, this can be used to avoid data

duplication when copying the entire file contents piecemeal, such as

when translating from one format to another. This was introduced to

convert between HDF4, HDF5, and XML. In the last couple years, NeXus

has agreed to only support HDF5 going forward.

I'll see if I can get to the bottom of the NeXpy problem you pointed out.

Pete

On 10/6/2014 6:48 PM, Mark Rivers wrote:

Hi Pete,

I've attached the file. I have now managed to get the hard link code running on the HDF5 plugin, so the data appears in 2 places:

/entry/data/data This plots OK

/entry/instrument/detector/data This gives the error I sent previously

You said previously:

One such program is "NeXpy". It is a Python-based GUI. A version can

run from /APSshare:

/APSshare/epd/rh6-x86_64/bin/nexpy

If that class path does not exist, nexpy won't plot the data.

Using the model I saw in the new HDF5 plugin, by adding the hard link,

nexpy should be able to plot the data.

I interpreted this to mean that it would plot that dataset if I simply did "Data/Plot Data" without selecting any data source, i.e. it would use /entry/data/data as the default. That does not work, I get a Python error.

I also have a question about the output of h5dump:

This is the API for H5Lcreate_hard from the HDF documentation:

---

Name: H5Lcreate_hard Signature: herr_t H5Lcreate_hard( hid_t obj_loc_id, const char *obj_name, hid_t link_loc_id, const char *link_name, hid_t lcpl_id, hid_t lapl_id )

Purpose: Creates a hard link to an object.

Description: H5Lcreate_hard creates a new hard link to a pre-existing object in an HDF5 file. The new link may be one of many that point to that object.

The target object must already exist in the file.

obj_loc_id and obj_name specify the location and name, respectively, of the target object, i.e., the object that the new hard link points to.

link_loc_id and link_name specify the location and name, respectively, of the new hard link.

---

So the first 2 arguments refer to the link target (i.e. the real object already in the file), and the second 2 are the new link.

This is my code that creates the hard link:

   herr_t err = H5Lcreate_hard(this->file, targetName.c_str(), this->file, linkName.c_str(), 0, 0);

printf("%s::%s called H5Lcreate_hard, target=%s, link=%s\n",

driverName, functionName, targetName.c_str(), linkName.c_str());

This is the output when I run the program.

NDFileHDF5::createHardLinks called H5Lcreate_hard, target=/entry/instrument/detector/data, link=/entry/data/data

So my real object is /entry/instrument/detector/data and my link is /entry/data/data

However, when I run h5dump on the file it looks like the opposite:

corvette:~/scratch>h5dump --header hdf5_link_xml_file_010.h5

HDF5 "hdf5_link_xml_file_010.h5" {

GROUP "/" {

GROUP "entry" {

   ATTRIBUTE "NX_class" {

      DATATYPE  H5T_STRING {

            STRSIZE 7;

            STRPAD H5T_STR_NULLTERM;

            CSET H5T_CSET_ASCII;

            CTYPE H5T_C_S1;

         }

      DATASPACE  SCALAR

   }

   GROUP "data" {

      ATTRIBUTE "NX_class" {

         DATATYPE  H5T_STRING {

               STRSIZE 6;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

         DATASPACE  SCALAR

      }

      DATASET "data" {

         DATATYPE  H5T_STD_U8LE

         DATASPACE  SIMPLE { ( 1024, 1024 ) / ( 1024, 1024 ) }

         ATTRIBUTE "NX_class" {

            DATATYPE  H5T_STRING {

                  STRSIZE 3;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "signal" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SCALAR

         }

      }

   }

   GROUP "instrument" {

      ATTRIBUTE "NX_class" {

         DATATYPE  H5T_STRING {

               STRSIZE 12;

               STRPAD H5T_STR_NULLTERM;

               CSET H5T_CSET_ASCII;

               CTYPE H5T_C_S1;

            }

         DATASPACE  SCALAR

      }

      GROUP "NDAttributes" {

         ATTRIBUTE "CameraManufacturer" {

            DATATYPE  H5T_STRING {

                  STRSIZE 18;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "CameraModel" {

            DATATYPE  H5T_STRING {

                  STRSIZE 15;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "Gettysburg" {

            DATATYPE  H5T_STRING {

                  STRSIZE 42;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "ID_Energy_EGU" {

            DATATYPE  H5T_STRING {

                  STRSIZE 3;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "NX_class" {

            DATATYPE  H5T_STRING {

                  STRSIZE 12;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "RingCurrent_EGU" {

            DATATYPE  H5T_STRING {

                  STRSIZE 2;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         ATTRIBUTE "hostname" {

            DATATYPE  H5T_STRING {

                  STRSIZE 25;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         DATASET "AcquireTime" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 19;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 23;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "E" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 17;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 12;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "ID_Energy" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 16;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 11;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "ImageCounter" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 13;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 13;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "MaxSizeX" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 15;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 10;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "MaxSizeY" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 15;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 10;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "Pi" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 11;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 12;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "RingCurrent" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 20;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 13;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "Ten" {

            DATATYPE  H5T_STD_I32LE

            DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

            ATTRIBUTE "description" {

               DATATYPE  H5T_STRING {

                     STRSIZE 8;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            ATTRIBUTE "source" {

               DATATYPE  H5T_STRING {

                     STRSIZE 12;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

         }

         DATASET "timestamp" {

            DATATYPE  H5T_IEEE_F64LE

            DATASPACE  SIMPLE { ( 1, 5 ) / ( 1, 5 ) }

         }

      }

      GROUP "detector" {

         ATTRIBUTE "NX_class" {

            DATATYPE  H5T_STRING {

                  STRSIZE 10;

                  STRPAD H5T_STR_NULLTERM;

                  CSET H5T_CSET_ASCII;

                  CTYPE H5T_C_S1;

               }

            DATASPACE  SCALAR

         }

         GROUP "NDAttributes" {

            ATTRIBUTE "NX_class" {

               DATATYPE  H5T_STRING {

                     STRSIZE 12;

                     STRPAD H5T_STR_NULLTERM;

                     CSET H5T_CSET_ASCII;

                     CTYPE H5T_C_S1;

                  }

               DATASPACE  SCALAR

            }

            DATASET "ColorMode" {

               DATATYPE  H5T_STD_I32LE

               DATASPACE  SIMPLE { ( 10 ) / ( 10 ) }

               ATTRIBUTE "description" {

                  DATATYPE  H5T_STRING {

                        STRSIZE 10;

                        STRPAD H5T_STR_NULLTERM;

                        CSET H5T_CSET_ASCII;

                        CTYPE H5T_C_S1;

                     }

                  DATASPACE  SCALAR

               }

               ATTRIBUTE "source" {

                  DATATYPE  H5T_STRING {

                        STRSIZE 6;

                        STRPAD H5T_STR_NULLTERM;

                        CSET H5T_CSET_ASCII;

                        CTYPE H5T_C_S1;

                     }

                  DATASPACE  SCALAR

               }

            }

         }

         DATASET "data" {

            HARDLINK "/entry/data/data"

         }

      }

      GROUP "performance" {

      }

   }

}

}

}

Note that /entry/data/data looks like the real array, and /entry/instrument/detector/data looks like it is just a HARDLINK?

What will happen if I put a second hardlink to the same data? Will it show up looking like data or like a link?

Thanks,

Mark

-----Original Message-----

From: Pete Jemian [mailto:jemian@anl.gov]

Sent: Monday, October 06, 2014 5:26 PM

To: Mark Rivers

Cc: Ray Osborn

Subject: Fwd: RE: nxvalidate?

Mark:

From your error report below, NeXpy has tried to examine your

NDattributes folder for the "shape" of the data to be plotted. Your

data is in:

/entry/instrument/detector/data

Your NDattributes folder is a sibling of the "data" field.

This error is produced in the NeXpy file

/src/nexpy/api/nexus/tree.py

in the NXfield class, getattr() method, where it is trying to find

numpy's ndarray attributes (note the similarity in names here). "shape"

is one attribute of a numpy ndarray.

NeXpy has a problem to be resolved in either the tree API or the caller

of that API when plotting this data (by double-clicking the

/entry/instrument/detector/data field).

Can you provide a test data file?

Pete

-------- Original Message --------

Subject: RE: nxvalidate?

Date: Mon, 6 Oct 2014 22:01:11 +0000

From: Mark Rivers <rivers@cars.uchicago.edumailto:rivers@cars.uchicago.edu>

To: 'Pete Jemian' <jemian@anl.govmailto:jemian@anl.gov>

That worked.

However, when I try to view images created with areaDetector I get the

attached errors for the attached file format.

Hovering over /entry/instrument/detector/data shows that it is

uint8(1024x1024), but when I double click I get the error:

"shape not in NXcollection:NDAttributes"

What's wrong?

Mark

-----Original Message-----

From: Pete Jemian [mailto:jemian@anl.gov]

Sent: Monday, October 06, 2014 3:23 PM

To: Mark Rivers

Subject: Re: nxvalidate?

PySide has its own Qt installation. Merging these is beyond my

experience. Instead, I have installed a separate Python and added

PySide to that.

Better use of your time to try an alternative way. Here's another

Python to try. I just booted that one on usaxscontrol2.cars.aps.anl.gov

and it starts with no troubles.

/APSshare/anaconda/x86_64/bin/nexpy

On 10/6/2014 3:19 PM, Mark Rivers wrote:

I want to use the Qt installation in /usr/local/Trolltech/Qt-4.8.4/, rather than my system install in /usr/lib64/, which is Qt-4.7.4. How do I do that?

-----Original Message-----

From: Pete Jemian [mailto:jemian@anl.gov]

Sent: Monday, October 06, 2014 3:14 PM

To: Mark Rivers

Subject: Re: nxvalidate?

Looks like the libraries from your /usr/local installation of Qt are

being found before the libraries in /usr/lib64/ needed to install PySide.

On 10/6/2014 12:25 PM, Mark Rivers wrote:

I do have Qt and qmake installed, and qmake is in my PATH:

corvette:~/scratch/NeXpy-0.4.5>which qmake

/usr/local/Trolltech/Qt-4.8.4/bin//qmake

---

Pete R. Jemian, Ph.D. <jemian@anl.govmailto:jemian@anl.gov>

Beam line Controls and Data Acquisition, Group Leader

Advanced Photon Source, Argonne National Laboratory

Argonne, IL 60439 630 - 252 - 3189

---

Education is the one thing for which people

   are willing to pay yet not receive.

---

From: Ulrik Kofoed Pedersen [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 9:50 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML layout (#31)

Just a quick comment to let you know that I have discussed this funny behaviour of a hardlinks with my colleague Tobias Richter who has a lot of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the main dataset when viewed with h5ls or h5dump. Tobias confirmed that this is indeed a behaviour that he has noticed when working with NeXus/HDF5 files.

We think it maybe due to datasets and hardlinks sharing the same objectID (like inodes on a filesystem). The applications h5dump and h5ls will iterate through the files objectIDs and will probably select the first one it finds as the 'main' dataset - with all others of the same objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood are just references to file system locations - apparently even if you delete the main dataset, but still have a hardlink, pointing to it, the real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-58196531.

[prjemian]

HDF5 hard links are common.
When data is first stored, a data object is created (such as a 2-D array
of 16-bit integers, 1k x 1k). At the same time, a reference (hard link)
to this data object is created as a directory entry (such as
/entry/instrument/detector/data). When a file writer chooses to create
a "hard link", they first start with a directory entry and then name the
new directory entry to be created (such as /entry/data/data). This
request creates a new directory reference to the same data object. To
the casual browser of the HDF5 file, the directory entries look
identical without regard to which one created the data object and which
one was the added link.

This is easy to see using HdfView. (The file is the same as yesterday
but modified by Ray Osborn to add data axes for plotting.) The data
object is stored at offset (11336,2) as viewed from either
/entry/instrument/detector/data
or /entry/data/data.

NeXus encountered the same situation. This could lead to data
duplication, such as when translating the content of a data file between
HDF5, HDF4, and XML. To avoid data duplication, NeXus adds an attribute
target="/HDF5/path/to/source/directory/entry" when creating the hard
link for the first time. Such a tag can be used to compare a directory
entry's name with this target attribute. If they do not match, do not
duplicate this data.

Here is python code (using h5py) to create an HDF5 link in a NeXus data
file:

def makeLink(parent, sourceObject, targetName):
     """
     create an internal NeXus (hard) link in an HDF5 file

     :param obj parent: parent group of source
     :param obj sourceObject: existing HDF5 object
     :param str targetName: HDF5 node path to be created,
                             such as ``/entry/data/data``
     """
     if not 'target' in sourceObject.attrs:
         # NeXus link, NOT an HDF5 link!
         sourceObject.attrs["target"] = str(sourceObject.name)
     import h5py.h5g
     parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

On 10/7/2014 9:49 AM, Ulrik Kofoed Pedersen wrote:

Just a quick comment to let you know that I have discussed this funny
behaviour of a hardlinks with my colleague Tobias Richter who has a lot
of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the
main dataset when viewed with h5ls or h5dump. Tobias confirmed that this
is indeed a behaviour that he has noticed when working with NeXus/HDF5
files.

We think it maybe due to datasets and hardlinks sharing the same
objectID (like inodes on a filesystem). The applications h5dump and h5ls
will iterate through the files objectIDs and will probably select the
first one it finds as the 'main' dataset - with all others of the same
objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood
are just references to file system locations - apparently even if you
delete the main dataset, but still have a hardlink, pointing to it, the
real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHub
#31 (comment).

[MarkRivers]

My current implementation uses syntax like this

  <hardlink name="data" source="/entry/instrument/detector/data"></hardlink>

It thus uses the XML attribute “source”.

The HDF5 documentation (and the Linux man pages for the “ln” command) both call the object being pointed to the “target”, not the “source”. Thus I am proposing to change the syntax to:

  <hardlink name="data" target="/entry/instrument/detector/data"></hardlink>

Comments?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:10 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML layout (#31)

HDF5 hard links are common.
When data is first stored, a data object is created (such as a 2-D array
of 16-bit integers, 1k x 1k). At the same time, a reference (hard link)
to this data object is created as a directory entry (such as
/entry/instrument/detector/data). When a file writer chooses to create
a "hard link", they first start with a directory entry and then name the
new directory entry to be created (such as /entry/data/data). This
request creates a new directory reference to the same data object. To
the casual browser of the HDF5 file, the directory entries look
identical without regard to which one created the data object and which
one was the added link.

This is easy to see using HdfView. (The file is the same as yesterday
but modified by Ray Osborn to add data axes for plotting.) The data
object is stored at offset (11336,2) as viewed from either
/entry/instrument/detector/data
or /entry/data/data.

NeXus encountered the same situation. This could lead to data
duplication, such as when translating the content of a data file between
HDF5, HDF4, and XML. To avoid data duplication, NeXus adds an attribute
target="/HDF5/path/to/source/directory/entry" when creating the hard
link for the first time. Such a tag can be used to compare a directory
entry's name with this target attribute. If they do not match, do not
duplicate this data.

Here is python code (using h5py) to create an HDF5 link in a NeXus data
file:

def makeLink(parent, sourceObject, targetName):
"""
create an internal NeXus (hard) link in an HDF5 file

:param obj parent: parent group of source
:param obj sourceObject: existing HDF5 object
:param str targetName: HDF5 node path to be created,
such as ``/entry/data/data``
"""
if not 'target' in sourceObject.attrs:
# NeXus link, NOT an HDF5 link!
sourceObject.attrs["target"] = str(sourceObject.name)
import h5py.h5g
parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

On 10/7/2014 9:49 AM, Ulrik Kofoed Pedersen wrote:

Just a quick comment to let you know that I have discussed this funny
behaviour of a hardlinks with my colleague Tobias Richter who has a lot
of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the
main dataset when viewed with h5ls or h5dump. Tobias confirmed that this
is indeed a behaviour that he has noticed when working with NeXus/HDF5
files.

We think it maybe due to datasets and hardlinks sharing the same
objectID (like inodes on a filesystem). The applications h5dump and h5ls
will iterate through the files objectIDs and will probably select the
first one it finds as the 'main' dataset - with all others of the same
objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood
are just references to file system locations - apparently even if you
delete the main dataset, but still have a hardlink, pointing to it, the
real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHub
#31 (comment).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-58199957.

[prjemian]

The HDF5 interface creates confusion by using "target" in this way.
Perhaps "existing" is a synonym that does not confuse.

On 10/7/2014 10:18 AM, Mark Rivers wrote:

My current implementation uses syntax like this

It thus uses the XML attribute “source”.

The HDF5 documentation (and the Linux man pages for the “ln” command)
both call the object being pointed to the “target”, not the “source”.
Thus I am proposing to change the syntax to:

Comments?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:10 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML
layout (#31)

HDF5 hard links are common.
When data is first stored, a data object is created (such as a 2-D array
of 16-bit integers, 1k x 1k). At the same time, a reference (hard link)
to this data object is created as a directory entry (such as
/entry/instrument/detector/data). When a file writer chooses to create
a "hard link", they first start with a directory entry and then name the
new directory entry to be created (such as /entry/data/data). This
request creates a new directory reference to the same data object. To
the casual browser of the HDF5 file, the directory entries look
identical without regard to which one created the data object and which
one was the added link.

This is easy to see using HdfView. (The file is the same as yesterday
but modified by Ray Osborn to add data axes for plotting.) The data
object is stored at offset (11336,2) as viewed from either
/entry/instrument/detector/data
or /entry/data/data.

NeXus encountered the same situation. This could lead to data
duplication, such as when translating the content of a data file between
HDF5, HDF4, and XML. To avoid data duplication, NeXus adds an attribute
target="/HDF5/path/to/source/directory/entry" when creating the hard
link for the first time. Such a tag can be used to compare a directory
entry's name with this target attribute. If they do not match, do not
duplicate this data.

Here is python code (using h5py) to create an HDF5 link in a NeXus data
file:

def makeLink(parent, sourceObject, targetName):
"""
create an internal NeXus (hard) link in an HDF5 file

:param obj parent: parent group of source
:param obj sourceObject: existing HDF5 object
:param str targetName: HDF5 node path to be created,
such as ``/entry/data/data``
"""
if not 'target' in sourceObject.attrs:
# NeXus link, NOT an HDF5 link!
sourceObject.attrs["target"] = str(sourceObject.name)
import h5py.h5g
parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

On 10/7/2014 9:49 AM, Ulrik Kofoed Pedersen wrote:

Just a quick comment to let you know that I have discussed this funny
behaviour of a hardlinks with my colleague Tobias Richter who has a lot
of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the
main dataset when viewed with h5ls or h5dump. Tobias confirmed that this
is indeed a behaviour that he has noticed when working with NeXus/HDF5
files.

We think it maybe due to datasets and hardlinks sharing the same
objectID (like inodes on a filesystem). The applications h5dump and h5ls
will iterate through the files objectIDs and will probably select the
first one it finds as the 'main' dataset - with all others of the same
objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood
are just references to file system locations - apparently even if you
delete the main dataset, but still have a hardlink, pointing to it, the
real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHub

#31 (comment).

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/31#issuecomment-58199957.

—
Reply to this email directly or view it on GitHub
#31 (comment).

[MarkRivers]

“target” it is the same syntax that the Linux man page for “ln” uses:

LN(1) User Commands LN(1)

NAME
ln - make links between files

SYNOPSIS
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
ln [OPTION]... TARGET (2nd form)
ln [OPTION]... TARGET... DIRECTORY (3rd form)
ln [OPTION]... -t DIRECTORY TARGET... (4th form)

DESCRIPTION
In the 1st form, create a link to TARGET with the name LINK_NAME. In the 2nd form, create a link to TARGET in the current directory. In the 3rd and 4th forms, create links
to each TARGET in DIRECTORY. Create hard links by default, symbolic links with --symbolic. When creating hard links, each TARGET must exist. Symbolic links can hold arbi-
trary text; if later resolved, a relative link is interpreted in relation to its parent directory.

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:29 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML layout (#31)

The HDF5 interface creates confusion by using "target" in this way.
Perhaps "existing" is a synonym that does not confuse.

On 10/7/2014 10:18 AM, Mark Rivers wrote:

My current implementation uses syntax like this

It thus uses the XML attribute “source”.

The HDF5 documentation (and the Linux man pages for the “ln” command)
both call the object being pointed to the “target”, not the “source”.
Thus I am proposing to change the syntax to:

Comments?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:10 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML
layout (#31)

HDF5 hard links are common.
When data is first stored, a data object is created (such as a 2-D array
of 16-bit integers, 1k x 1k). At the same time, a reference (hard link)
to this data object is created as a directory entry (such as
/entry/instrument/detector/data). When a file writer chooses to create
a "hard link", they first start with a directory entry and then name the
new directory entry to be created (such as /entry/data/data). This
request creates a new directory reference to the same data object. To
the casual browser of the HDF5 file, the directory entries look
identical without regard to which one created the data object and which
one was the added link.

This is easy to see using HdfView. (The file is the same as yesterday
but modified by Ray Osborn to add data axes for plotting.) The data
object is stored at offset (11336,2) as viewed from either
/entry/instrument/detector/data
or /entry/data/data.

NeXus encountered the same situation. This could lead to data
duplication, such as when translating the content of a data file between
HDF5, HDF4, and XML. To avoid data duplication, NeXus adds an attribute
target="/HDF5/path/to/source/directory/entry" when creating the hard
link for the first time. Such a tag can be used to compare a directory
entry's name with this target attribute. If they do not match, do not
duplicate this data.

Here is python code (using h5py) to create an HDF5 link in a NeXus data
file:

def makeLink(parent, sourceObject, targetName):
"""
create an internal NeXus (hard) link in an HDF5 file

:param obj parent: parent group of source
:param obj sourceObject: existing HDF5 object
:param str targetName: HDF5 node path to be created,
such as ``/entry/data/data``
"""
if not 'target' in sourceObject.attrs:
# NeXus link, NOT an HDF5 link!
sourceObject.attrs["target"] = str(sourceObject.name)
import h5py.h5g
parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

On 10/7/2014 9:49 AM, Ulrik Kofoed Pedersen wrote:

Just a quick comment to let you know that I have discussed this funny
behaviour of a hardlinks with my colleague Tobias Richter who has a lot
of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear as the
main dataset when viewed with h5ls or h5dump. Tobias confirmed that this
is indeed a behaviour that he has noticed when working with NeXus/HDF5
files.

We think it maybe due to datasets and hardlinks sharing the same
objectID (like inodes on a filesystem). The applications h5dump and h5ls
will iterate through the files objectIDs and will probably select the
first one it finds as the 'main' dataset - with all others of the same
objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the hood
are just references to file system locations - apparently even if you
delete the main dataset, but still have a hardlink, pointing to it, the
real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHub

#31 (comment).

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/31#issuecomment-58199957.

—
Reply to this email directly or view it on GitHub
#31 (comment).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/31#issuecomment-58203131.

[prjemian]

Agreed, it is a matter of perspective. Documentation can help set that
perspective. "Creating a link from this new entry back to the existing
TARGET entry"

On 10/7/2014 10:33 AM, Mark Rivers wrote:

“target” it is the same syntax that the Linux man page for “ln” uses:

LN(1) User Commands LN(1)

NAME
ln - make links between files

SYNOPSIS
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
ln [OPTION]... TARGET (2nd form)
ln [OPTION]... TARGET... DIRECTORY (3rd form)
ln [OPTION]... -t DIRECTORY TARGET... (4th form)

DESCRIPTION
In the 1st form, create a link to TARGET with the name LINK_NAME. In the
2nd form, create a link to TARGET in the current directory. In the 3rd
and 4th forms, create links
to each TARGET in DIRECTORY. Create hard links by default, symbolic
links with --symbolic. When creating hard links, each TARGET must exist.
Symbolic links can hold arbi-
trary text; if later resolved, a relative link is interpreted in
relation to its parent directory.

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:29 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML
layout (#31)

The HDF5 interface creates confusion by using "target" in this way.
Perhaps "existing" is a synonym that does not confuse.

On 10/7/2014 10:18 AM, Mark Rivers wrote:

My current implementation uses syntax like this

It thus uses the XML attribute “source”.

The HDF5 documentation (and the Linux man pages for the “ln” command)
both call the object being pointed to the “target”, not the “source”.
Thus I am proposing to change the syntax to:

Comments?

Mark

From: Pete R Jemian [mailto:notifications@github.com]
Sent: Tuesday, October 07, 2014 10:10 AM
To: areaDetector/ADCore
Cc: Mark Rivers
Subject: Re: [ADCore] HDF5 plugin to support defining hard links in XML
layout (#31)

HDF5 hard links are common.
When data is first stored, a data object is created (such as a 2-D array
of 16-bit integers, 1k x 1k). At the same time, a reference (hard link)
to this data object is created as a directory entry (such as
/entry/instrument/detector/data). When a file writer chooses to create
a "hard link", they first start with a directory entry and then name the
new directory entry to be created (such as /entry/data/data). This
request creates a new directory reference to the same data object. To
the casual browser of the HDF5 file, the directory entries look
identical without regard to which one created the data object and which
one was the added link.

This is easy to see using HdfView. (The file is the same as yesterday
but modified by Ray Osborn to add data axes for plotting.) The data
object is stored at offset (11336,2) as viewed from either
/entry/instrument/detector/data
or /entry/data/data.

NeXus encountered the same situation. This could lead to data
duplication, such as when translating the content of a data file between
HDF5, HDF4, and XML. To avoid data duplication, NeXus adds an attribute
target="/HDF5/path/to/source/directory/entry" when creating the hard
link for the first time. Such a tag can be used to compare a directory
entry's name with this target attribute. If they do not match, do not
duplicate this data.

Here is python code (using h5py) to create an HDF5 link in a NeXus data
file:

def makeLink(parent, sourceObject, targetName):
"""
create an internal NeXus (hard) link in an HDF5 file

:param obj parent: parent group of source
:param obj sourceObject: existing HDF5 object
:param str targetName: HDF5 node path to be created,
such as ``/entry/data/data``
"""
if not 'target' in sourceObject.attrs:
# NeXus link, NOT an HDF5 link!
sourceObject.attrs["target"] = str(sourceObject.name)
import h5py.h5g
parent._id.link(sourceObject.name, targetName, h5py.h5g.LINK_HARD)

On 10/7/2014 9:49 AM, Ulrik Kofoed Pedersen wrote:

Just a quick comment to let you know that I have discussed this funny
behaviour of a hardlinks with my colleague Tobias Richter who has a
lot
of NeXus/HDF5 experience.

What confused me was that the hardlink in my example would appear
as the
main dataset when viewed with h5ls or h5dump. Tobias confirmed that
this
is indeed a behaviour that he has noticed when working with NeXus/HDF5
files.

We think it maybe due to datasets and hardlinks sharing the same
objectID (like inodes on a filesystem). The applications h5dump and
h5ls
will iterate through the files objectIDs and will probably select the
first one it finds as the 'main' dataset - with all others of the same
objectID, being labelled as references...

It really doesn't make any difference as all the datasets under the
hood
are just references to file system locations - apparently even if you
delete the main dataset, but still have a hardlink, pointing to it,
the
real data is not deleted (or lost) from disk.

—
Reply to this email directly or view it on GitHub

#31 (comment).

—
Reply to this email directly or view it on

GitHubhttps://github.com//issues/31#issuecomment-58199957.

—
Reply to this email directly or view it on GitHub

#31 (comment).

—
Reply to this email directly or view it on
GitHubhttps://github.com//issues/31#issuecomment-58203131.

—
Reply to this email directly or view it on GitHub
#31 (comment).