Ortho & LiDAR Accuracy Webinar with Mike Tully & Charles O'Hara

Just another WordPress site

Ortho & LiDAR Accuracy Webinar with Mike Tully & Charles O'Hara

welcome to aerial services also and light our accuracy webinar number two of three and the arrow services summer webinar series this is your host Josh on McNary marketing manager at aerial services we are pleased you have joined us and hope you walk away having learned about how ortho and letter accuracy can be improved for your geospatial projects but first a few housekeeping notes you may ask questions via the questions pane on your screen we’ll collect these questions for your spots from our speakers at the end of the presentation we want your feedback content us via the chat pane or the post webinar survey with any ideas questions or concerns regarding this webinar the post webinar survey will appear on your screen after you read the session please use that as an opportunity to share your feedback lastly but not least not that aerial services will have one more webinar as part of our summer series UAV and geospatial will take place on August 22nd please join us visit www.ovionmedia.com or you can purchase echo by analysts president CEO Chuck O’Hara who will share their expert knowledge with you today Chuck and Mike have been on a mission in the last year to help educate geospatial users about the do’s and don’ts of using accurate or inaccurate geo data they spoke together on this topic at last spring SPR annual conference in Sacramento and are scheduled to do a follow up session at this fall’s joint ASPRS slash maps 2012 specialty conference in Tampa their insights into procuring and checking geospatial data to ensure accurate solutions analysis has tactics sessions in the past and today’s webinar attendance is continued proof of the relevancy of these topics so with that I’m happy to turn the session over to my friends Mike and Chuck we’ll get on with the show Mike will you lead us off sure thanks Josh thank you everybody for joining us today so let me ask you if I was to put you all against the back wall and begin throwing darts at you would you rather that I be imprecise or inaccurate the interesting question and I’m not going to answer it right now but we’ll talk about that and a lot of other aspects of accuracy today in just a bit next slide unless you’re from Texas you probably notice something wrong with this ma’am and we’ve all seen them bad maps there’s this one or the next slide labeling might be a problem scale might be a problem next slide sometimes it’s just the underlying data that causes a problem like this dim that was used to create an ortho or the next slide another common type of problem with underlying data that we see commonly in geospatial products next slide sometimes our our vectors don’t line up and when this happens what do we do are can you tell if the roads the vector roads are bad or are they is the Oracle bad and how do you know which data set is inaccurate there could be a problem of projections maybe a transformation was a problem may be the vector roads are bad you just don’t know unless you know more about accuracy so today next slide what we’d like you to take away from this is why you should care about the accuracy of your geospatial data sets today we find ourselves you know awash with free or inexpensive geospatial data orthos and lidar even you know ipad apps the amount of data available for use in our geospatial applications is and projects is going to continue to explode well into the future and if you don’t have a professional understanding of and care about accuracy you may harm yourself or others with that data

next slide we’re all professionals and we don’t ever want this said about us the next slide Josh yeah thank you so we all want that said about us so you know what do we mean by accurate how do I know how accurate my geospatial data is how accurate is accurate we want to talk about those things today um next slide the professional use of geospatial data starts with metadata you can’t know the accuracy of a data set without first having it’s metadata it is not uncommon for Austin Chuck when we visit clients using geospatial data that they don’t have any knowledge of the metadata associated with the data sets they’re using they don’t know where it originated they don’t know who created it they don’t know how it was created and they often don’t know why it was created metadata properly structured metadata hopefully fgdc compliant metadata answers all of those types of questions and and is where you start understanding accuracy next slide um one of the things hopefully that you’ll find in good metadata is a description about the intended use of that data a professional should be very concerned about the intended use of a geospatial data set because of the threat to public welfare and safety if it’s if that data is used improperly data is not to be used for some things for some applications for example using 2009 Tiger data Road centreline data that has a 25-foot horizontal accuracy or that guarantees 25-foot horizontal accuracy can be used that to align your property boundaries – you may not have a good idea if you’re expecting similar ortho accuracies in the range of 12 inches so in fact uh if you’ve ever looked the metadata for a tiger data says quote no warranty expressed or implied is made with regard to the accuracy of the data and the tiger shapefile and no liability is assumed as to the positional or attribute accuracy of the data so know what the intended use of the data is because you can very easily misuse it and think it’s more accurate than it is or was ever intended to be you know what’s the positional accuracy of Google Earth if you look though you’ll find that there is no accuracy defined on their Wikipedia page and it’s left up to do it as an unknown so maps are created for a reason they can be used for some purposes but not all of them so be very careful and understand their intended use next slide speaking of accuracy another common misconception in the GIS community that we run into a lot is is that accuracy can be in by GSD here the pixel size or when we’re talking about wide art that people assume that lidar is more accurate with more points on the ground than less and I want to use this picture here to illustrate how this assumption is flatly untrue there is no direct relationship between resolution and accuracy so on the left here is a picture the same picture is on the right the picture on the left is high-resolution it’s broke into 36 pixels so it’s high res 36 pixels the one on the right is just 4 pixels to low res got a very big GSD because both pictures are from the same ortho they all have the exact same accuracy the higher resolution on the left is no more accurate than low resolution there’s no direct

relationship between accuracy and resolution I could like why I showed you lidar data that has a high point count per square meter and others that with a low point count you cannot infer accuracy based on the density of lidar on the ground again you’ll discover this kind of information in well-written metadata it’s important to understand this next slide Josh um so let’s get more technical here for uh per minute and discuss err and accuracy any time we measure something no matter how careful we are matter what tools we use that measurement has err there’s no way to escape it a variety of factors contributed that air and what this graph is showing us is that of all the air that total air budget most of it is systemic air and a very small part of his random error bias err systematic error we can do something we can more carefully measure we can use better instruments we can model away air it’s systemic air the random air the bias there is it’s a different type of air that really is just inherent in the instrument we use and we really can’t model it out and it’s just part of measurements so for example if and then sorry down below we exploded that systemic air to show that in a typical remote sensing application whether it’s lidar or camera there’s basically four types of air that making up systemic air again air that we can do something about by far the biggest source of air in our remote sense data would be position or GPS data and attitude which way the sensor is pointing other types of air would include instrument air air is associate with the camera itself the lens inside the camera the mirror inside the laser things like that data processing is another source of air we can we can introduce air just by the way we process data the software we use the the projection we apply the transformations the tonal adjustments that we might make and then lastly a you know human interpretation of the data we might classify things wrong we might make mistakes in the way we identify which features operator training poor operating training can can influence air in that respect so um it’s the photogrammetry rest’s or the remote sensing experts responsibility to understand these sources of error and manage them responsibly we’re gonna do that with our training with designing the project with knowledge about the parameters and and such to control or reduce the airs that are possible given the project restraints next slice Josh from the air sources we move on to the standards the accuracy standards of measuring and reporting error this is a list of the different standards we’ve all most of us have heard of by far today the NSS ba is the predominant standard in use and this is how as you acquire or produce geospatial data this is how act positional accuracy should be defined for digital mapping data older standards that are still used some are ASPRS and national map accuracy standards and of course there are other organizations that have their own standards like FEMA NOAA the US Corps of Engineers they might have other standards you need to them I will learn here in a little bit that ASPRS is coming out with a new some new revised standards that may actually replace or supplement an SSD a but for now an FDA is the private the most

logical standard to use for geospatial data they’re important because when used properly they objectively quantify and report the positional accuracy of a data set and this is important because it helps us compare one data set with another if the same standard is used to measure and report a data set and you can compare those two data sets with each other and good metadata will report if the data set was tested for accuracy and to what accuracy standard it was tested to so that I’m gonna turn the rest over to Chuck Chuck take the big get away thank you so much Mike next slide please so when we’re talking about accuracy or or uncertainty it’s important that we understand certain concepts that include accuracy and that’s quantifiable accuracy versus uncertainty which is not quantified but it’s it’s a lack of knowledge about the characteristics of the data and when we’re talking about accuracy we’re talking about things such as precision the completeness of data the correctness of data the consistency of data the currency of the data meaning how up-to-date it is the lineage of it meaning how it was acquired processed and what modifications or transformations have been made to it and and other factors that can influence the accuracy and the quality of the data next slide please so when we’re when we’re looking at geospatial data methods that we use to know whether the data are good include visual inspection or what we would call a subjective or qualitative evaluation the evaluation of independent experts meaning having subject matter experts that come in independent of the production team or of the customer to help advise as to the quality and the accuracy of the data the use of demonstrated methods that assure that the data are being produced in methods that in the past have been demonstrated to deliver data that meet the accuracy specifications and requirements and the ability to deliver verify and report the accuracy of the results those are things that help us to ascertain whether or not the data are good enough or they’re fit for use next slide please so when it does count the things that we’re concerned with are measurement accuracy meaning how are we looking at at photo identifiable locations survey checkpoint locations and how do we compare that to locations or to positional information in the geospatial data the opinion of experts the use of independent firms to provide services that help us to verify the data or to collect survey check points or control or to go out and collect land use or land cover information and the use of standard and verifiable methods in a consistent and objective manner for the project those are all important factors to consider when when accuracy counts next slide please so in doing in doing this in practice many of the standards guidelines and specifications that have been in use or those that are new and evolving refer back to primary references such as the often cited works of Greene Walton Schultz back in 1962 so whether we’re looking at standards from the national standards for spatial data accuracy an SSD a or whether we’re looking at the new standards being published by USGS that we are referring to as the version 13 specification for lidar guidelines and base specification or whether we’re looking at references such as Congleton and green’s work on assessing the accuracy of remotely sense data most of these standards refer back to earlier works of the deal with the quantification of accuracy error and uncertainty and as our data evolved the producer and end-user communities of practitioners are seeing greater emphasis as Mike said earlier on high resolution products but the capabilities to quantify data as having high accuracy or high quality are based on standard

methods and governing accuracy equations next slide please so as we’re taking these standards in reducing them to application we’re faced with reducing accuracy standards two fundamental governing equations into providing methods that automate the input of the data to be tested methods to automate input of reference data comprising precisely survey checkpoints and quantification of checkpoint error and when we’re quantifying check put checkpoint error that that’s used to compute products such as the mean error standardized error which is which is also commonly referred to as standard deviation the root mean square error in XY and Z as well as the national standards for spatial data accuracy which typically computes the 95% confidence level and that’s used for both horizontal accuracy accuracy R as well as the vertical accuracy accuracy Z and we often see that referred to as the 95 percent compass level in SSD a accuracy so one other thing to be aware of is that when we’re looking at our MSE the value of the checkpoint data is typically acquired at an accuracy that’s three times the accuracy of the required accuracy of the verified product so if we’re looking at data that have a requirement for three inch accuracy we would have to be surveying data with a precision of one inch or better in accuracy next slide please Chuck can I add something there absolutely like yeah I just wanted to say that you know those formulas look daunting to the to some people and but they’re these are standard statistical formulas that are used to measure air and and assess air in almost any application you can imagine it’s not just geospatial so don’t let the formulas carrier they’re very standard there’s nothing unique here to geospatial these are just standard statistical formulas right and and it’s it’s the challenge of the practitioner to to basically eliminate the complexity by making a user-friendly application that allows the data to be input the checkpoints to be input and the checkpoint offsets to be extracted automatically allowing the the equations to be performed behind the scenes transparently in a very consistent and easily followed manner so those governing equations are behind the scenes you know in a well-defined and delivered application that automates the QA process so having said that the existing standards that that we’re accustomed to now might be supplemented or replaced by next generation accuracy standards so we’re in this age when we have data resolution from new digital sensors that are providing data of unprecedented resolution and along with that positioning in geo referencing technologies deliver the capability to process and deliver data of extraordinary accuracy so along with these capabilities for higher resolution and accuracy there’s a need for next-generation definitions and standards that define classes for data of higher accuracy so to meet this need we see new accuracy standards emerged for digital geospatial data from organizations like ASPRS for these new data there may be accuracy definitions for standard high accuracy lower accuracy and extra high accuracy and in the examples that follow Class A products refer to standard high accuracy data Class B refers to lower accuracy geospatial data and Class X products refer to extra high accuracy geospatial data that might be suitable for the most demanding user applications and I want to note that these are shown as examples and the actual final standards that get published by ASPRS might vary somewhat from what you’re going to see in these next slides next slide please so here we see ortho imagery horizontal accuracy standards these standards attempt to crosswalk the classless or scaleless image resolutions to NSS DEA specifications for a particular class of data accuracy and to extend that to an

equivalent National map accuracy standard specification and to appropriate map scale use now currently as Mike said earlier the NSS DEA standards are scaleless but here we attempt to specify our MSE and 95% confidence levels and to crosswalk those two scales for appropriate use of the data so in this table we can see that accuracy Class A has a root mean square error that’s equal to 1.5 times the pixel size Class B in which root mean square error is 2 times the pixel size and accuracy class X has an RMS e equal to the pixel size so if we look at class 1 X or class 1 extra high accuracy which is the top row we can see a pixel size of 3 inches our MSE and X are Y of 3 inches and an RMS er of 4.2 inches and in SSD accuracy of seven point three inches we see the equivalent si mass 90% confidence level and the appropriate scale and a new item that we see is the mosaic seam line maximum mismatch of six inches that’s a seam line mismatch of two pixels that’s that’s pretty high accuracy and we’ll need some pretty nice tools to be able to verify in QA that data can meet these kind of standards next slide please similarly for lidar we see standards in which point density rmse and Z fundamental vertical accuracy with the 95% confidence level consolidated vert vertical accuracy at the 95th percentile and and equivalent n NS in mas specifications and relative accuracy in the swath overlap are defined here we see that class 2a which is in the middle and highlighted equates to the current usgs version 13 specification in which we see a 1 meter pulses per square meter 12 point five centimeters our MSE in the Z and the Associated fundamental vertical accuracy and consolidated vertical axis that comply with the current USDA spec note note here that the relative accuracy in the swath overlap is cited as an RMS B value now the difference between rmsd and our MSE is important to note our MSE as we specified earlier is an accuracy that’s computed from a reference that’s three times more accurate than the governing accuracy requirement for the data set here our MSD is root-mean-square difference and this is a difference in values computed between two data sets that are expected to have similar accuracy so this is an important new new change that might be inserted into accuracy specifications for lidar data now if we look at this data all of these columns except fourth column two and the final column would also apply to digital elevation data compiled from stereophotogrammetry so if we want to look at new data that are being acquired in some areas such as in the Pacific Northwest data are being acquired that they might meet class 1x or eight pulses per square meter RMS II and the Z of 9.2 and soif accuracy in the overlap of of six centimeters next slide please so into this era of higher resolution and current data that have to be maintained we’re seeing a change in ortho resolutions from 1 meter or or lower resolution to 3 inch or 6 inch orthos a lot of the times these data are being compiled or acquired by state county and local government we’re seeing light our data that are being collected not at a 1 meter posting or five meter postings but with with pulse densities that approach or exceed 8 pulses per square meter as we look to use data across regions across States or across the US we have to look at whether we can simply blend the data together or do we do national level reviews of data specifications and accuracy to understand which data can be combined into national or regional products and the question becomes how do we look at standard methods and test

approaches and methods and metrics for making decisions about what data we can continue to use or can can use for broad scale or regional applications as well as how do we evaluate the best available data for local applications next slide please I’ll just add to that too that as we see this geospatial data becoming more and more dense higher and higher accuracy there’s you know more and more lidar points instead of millions of points there’s billions instead of hundreds of photographs there’s thousands or hundreds of thousands and how do you test the accuracy of a billion points or a hundred thousand photographs it begs an application that facilitates measuring the accuracy of that value that great value of data right and and that that goes to the issue of do you just check part of the data and you use your your manual methods or do you develop automated methods that will consistently and completely only test the data and that’s that’s where we’re going to to look at at several points as we move through the remaining slides but this slide positions the the method for actually conducting checkpoint analysis that’s used to measure positional accuracy and here we have a reference coordinate shown in blue and an image photo identified location shown in yellow and from this we compute the offsets in X&Y to calculate the error offset or checkpoint error and those terms are combined by taking the square and the square root of the sum of the squares to come up with a total radial offset of a given reference location to the image coordinate so this is the method whereby we compare referenced and photo interpreted locations in order to compute horizontal accuracy similarly we have reference locations that compute a Z value and in the vertical we compare that to the elevation from either the point cloud lidar data or to the D en surface and that’s a Z or Z offset next slide please as we as we take these checkpoints and we tabulate them for data collected across a study area where we’re we’re advised to collect 20 or more than 20 points in order to get a statistically rigorous sample set and when we tabulate those data accuracy statistics can be computed from the checkpoint offsets we identify the photo locations and compute those differences in X and Y’s compute the accuracy statistics and we use charts and graphs X to deliver understanding about the distribution the magnitude and the sources of error next slide please so as we’re looking at this when we look at the distribution of checkpoints across a study area we can look at a circular error plot that shows the amount of X&Y error contributed by each point and the top circular graphic shows a series of points that all fall in the northwest quadrant and well out of the circular center of the target this would indicate a data set with a large amount of systematic bias and similarly the plot that’s to the right of that or the vector offset plot shows a series of vectors all pointing to the north and west away from the the source of the point location for the error so here we see a data set that has a large amount of systematic bias when we remove that bias we’re left with error that may include other forms of data that other forms of error that can be removed and the result is a small amount of error that’s randomly distributed and has a near zero mean and at that point we have what we would consider to be good data so this shows how the circular plots vector offset plots and statistics can combine to give us understanding of have we removed bias have we addressed error and are we delivering data that meet to meet the accuracy requirements of specification next slide please so again this gets back to to the slide that Mike presented earlier in which systematic error comprise the largest portion of the error budget we can eliminate or address as much as possible

of that in calibrating the data getting our positioning information correct addressing any instrument issues calibration doing correct data processing making sure that our projection and datum information are correct and our our interpretation are the rules and the transforms that we apply to the data are done consistently and appropriately and after we’ve done that we’ve eliminated as much as we can from the error budget and there’s always going to be a certain amount of random error that’s left but reducing that to the minimum amount possible is all part of the process of creating highly accurate and high quality data next slide please so as we’re measuring checkpoint error and accuracy we get to the this concept of absolute accuracy and relative accuracy absolute accuracy as we’ve said is computed by comparing survey checkpoint locations that are three times the accuracy of the required actress of the data set to the points that are located in in the data set or to photo ID locations so this is highly accurate reference compared to position locations in the data set when we’re looking at relative accuracy we’re looking at the location of feature in a reference image data set compared to the location of the same feature and a test in its data set for the absolute accuracy we’re computing root mean square error for the relative accuracy we’re really computing a root mean squared difference products this goes back to this rmsd concept that was earlier pointed out in this next generation accuracy standards or criteria that we’ll be seeing in the future next slide please when we’re looking at checkpoint error and accuracy whether it’s absolute or relative for different sensor systems we may have situations where we have a rotary mounted or helicopter based ladder and ortho acquisition as well as as on fixed-wing craft in which we’re collecting or Co collecting optical or aerial imagery as well as lidar data we can use absolute accuracy methods or checkpoints to compute the horizontal accuracy of both products we can also look at at the same locations in both to to cross-check the data to assure that they’re there correctly Co registered so we can evaluate absolute accuracy as well as relative accuracy both between image types collected over time or with different resolutions or different spectral bands or in this case different sensor platforms we’re looking at imagery versus lidar intensity data next slide please as we evaluate lidar in digital elevation model absolute accuracy more and more often what we’re seeing are child data and data checks for a consistency and completeness of the delivery and an abundance of check points collected over multiple land cover types in this particular application there are approximately 15 counties of data and as many as as five land cover types and in each county of a requirement to collect 20 check points per land cover types so we may have over 800 points and in this particular case we’re looking at the bare earth land cover type and we have 271 observations for that the accuracy statistics that we would typically collect for those would include minimum Delta Z maximum mean number of observations and the standard deviation as well as root mean square error in the Z now since we’re evaluating this for the bare earth cat this is the fundamental the fundamental vertical accuracy and we apply the 95% confidence level in the computing Z value which is 1.96 times RMS easy or value of thirteen point seven thirteen point seven centimeters in this case next slide please on the previous slide what we saw were the final deliverable products and an accuracy checks performed on those but as we’re looking at the light our life cycle we have to look at it as life cycle products and if we don’t check verify and address issues in different phases of the project then it’s very likely that the data will fail at the end so as we’re doing acquisition calibration and final products it’s

important that we perform checks along the way in the acquisition phase we’re certainly interested in completeness of coverage adequacy of spacing density the consistency of the observations the amount of overlap and the data again the coverage and in every step we’re looking at the LA s compliance once where once we’ve calibrated the swath data we’re still looking at the completeness the la s compliance the degree to which the data calibration has delivered a relative accuracy that meets the requirement for a relative accuracy in the swath overlap and again the coverage once we get to the final products of the Declassified and tiled data we’re looking at the absolute accuracy of the DIMM and the LA s data the degree to which the data had been correctly hydro flattened the presence of brake lines as well as water body outlines the metadata that are to accompany the data and any lad our accuracy and surveying reports that are required to be provided along with the deliverables next slide please so as we’re looking at at verification across the lifecycle it’s important that we have tools where we can efficiently and consistently look at the coverage of the lidar swath data are we getting complete coverage or do our flat lines include voids or areas where we are missing and/or have gaps between our overflights if we have a requirement that there be 20% overlap do we have methods to efficiently verify that and in looking at the coverage as well as overlaps do we have methods to efficiently determine if there are holes or voids in the data and if there are are those due to water bodies which are allowable holes or all those holes or voids due to system outages or holidays in the system collection it’s important that we have methods where we can rapidly and efficiently determine coverage overlaps and holes or voids next slide please so as we look at that when we’re evaluating coverage it’s also important that we can count or evaluate the density and the spacing of the data according to the USGS version 13 draft specification there’s a requirement for coverage and spatial distribution that we have a pulse count grid that’s created at at nominal pulse spacing times two and we have at least one pulse count in at least 90% of the cells that are created that’s a threshold that we can use to determine pass or fail on density and spatial distribution and there is a need to be able to efficiently create raster pulse count density grids when we combine that with our holes data set and our coverage what we find is the ability to use those an overlap and be able to efficiently determine whether or not any gaps in the data are due to water or if there holds and this also can help us to begin to get a head start on creating or come piling our hydro brake lines and looking at areas where we would expect water bodies to be in our income and compiling and preparing our data for review and for preparation of hydro fat and DMS next slide please as we look at at the data as we move into the the second phase which is evaluating the calibration it’s important that we we understand that properly calibrated and adjusted swath data should result in high relative accuracy between the swaths now even if we meet the requirement to produce data that are are better than 10 centimeters in root mean square error between the swaths there’s still the possibility there could be visible step errors and if we if we compute sill shaded light our swath data and we look at those we shouldn’t be able to really see detectable step errors between lidar swaths so properly calibrated data such as are shown on the right would eliminate visible step errors between the lidar swaths as shown and would eliminate any kind of patchwork quilting also between the data sets next slide please as we look at lidar relative accuracy or error testing we evaluate the swath overlap and there are many methods that are being used to test this and in this case we are substituting or sieving

subdividing the overlap into a number of segments extracting the Z values from both la s files and computing the differences in Z values and here again we’re computing an RMS D in Z for relative accuracy statistics common methods that are also in uses is looking include looking at Z difference raster’s which are images that compute the difference between the latter elevations and are used to show visuals that depict the differences in increasing shades of green or red for increasing offset between the two different light our data sets so here on the right side we can see root mean square error in Z of 7.78 that’s better than the ten centimeters that are required for pass/fail of relative accuracy in the lidar data next slide please so it’s important I guess for us to look very you know at a high level of what’s important to test in the phases of the lidar life cycle in phase one where we’re looking at uncalibrated data we’re looking at coverage completeness correctness consistency gaps and density and those are primarily conducted on the producer side to answer the question am i done flying if those data are also delivered to the customer the customer can evaluate the data and say were these data collected correctly completely and consistently to give me the data I need in Phase two we’re looking at calibrated data and evaluating whether the data meets spec in Phase three we’re looking at whether the data meets specs on a retic ready for delivery next slide please over and above the quantitative accuracy verification there are advanced visual metrics and these are our metrics that are being used in large part by most folks that are doing quality assurance now and here we’re looking at whether we can see artifacts from the from the ten data in our hydro flattening we’re also able to see errors in the stream where there’s not a monotonic increase in the elevation or a smooth surface for the stream water body we also can see things like spikes or pits as well as improper bridge removal from the de M product this these are typically advanced visual metrics that are performed and this shows the raster data but we can also look at things like whether or not brake lines float above or there below the point cloud data if we’re looking at at the lidar point cloud and 3d as well as 3d brake lines next slide please so now we get to the question of who’s doing the work and increasingly we’re seeing a situation where there are federal state and local partnerships in those cases we have blended requirements for verification and that presents challenges to whoever is doing the verification and what we want to ensure is that we’re not just sacrificing quality assurance just to get more data in in truth whether whether we have the the case that we’re rounding up the usual suspects the best vendors even in those cases we need new methods to automate verification and new tools and methods are certainly needed to reduce the cost and increase the efficiency to fully verify and audit new data and and at the bottom you know the the thing that that’s trying to show is that if you if you only have a hammer everything ends up looking like a nail and the reality is that we probably need more tools that can address how we can verify and assure the quality of next generation data next slide please so in reality the status quo is that that almost all companies that are producing the data have accuracy and quality tests there is no easy button to this there is no single tool that does it all it’s a complex combination of tools and methods the good news is that there are standards and working groups that are providing advanced science technology and guidelines that can help us to assure the data to meet next generation requirements and certainly in in all cases within within companies there are existing internal or homegrown solutions scripts tools and methods that are manual and in many cases automated tests and certainly there are internal versus commercial off-the-shelf solutions that are evolving to provide industry standard practices next slide please if we’re looking internal versus commercial methods there’s a possibility that the internal

methods may may have some bias to their production methods whereas the commercial methods may be independent of any sensor system or production capabilities internal tools can become outdated and legacy whereas we would hope that the commercial tools provide new capabilities and are current certainly internal tools are self supported and that means that you have to keep them and whoever’s using that or providing that tool supported internally whereas with commercial tools they’re provided support provided externally with internal tools you just have internal users as opposed to commercial tools with communities of users and internal tools are expensive to develop and maintain whereas external tools may be purchased and maintained internal tools they’re of manual processing external tools automated if you have a large project in which there are multiple partners and participants if everyone has their own tools you can have inconsistency and results if everyone’s using the same methods you would expect to have consistent results internal tools are not easily shared whereas commercial tools that the products the tools and the methods are more easily shared so this is just some some pros and cons and some comparisons between internal versus cost tools for auditing data and as we look at the geodata status quo next slide please we see a situation where we have multi generation map products both in ortho as well as lidar data where we where acquisitions have been conducted over time with different contracts different data and different vendors and different verification vendors have different specifications and different verification approaches and the symptoms of that that may become a reality that we have to deal with is that the data can become more of a patchwork quilt of resolution sources and accuracies reacquisition may become a major trend and there may be too much data to just man check the reality is that if if we can’t check them all or manually verify them does that may mean that we’ve become prone to saying we’re going to check 20% of the tiles and what this means is that we may miss major fatal flaws in the data and it may also mean that we lack consistent methods and approaches that are applied uniformly across the data sets these are these are symptoms and issues that we have to be cognizant of as we as we produce news data next slide please and here I believe is where I turn it back over to Mike to address where best practices are needed and to discuss briefly what those are and how they vary okay we certainly need better tools and we see them being developed by folks like Chuck and spatial information systems solutions but things like densities the density of points on the ground coverage a consistent coverage across the project area overlap whether it’s orthos or lidar data have to be consistent sensors need to be calibrated typically before each mission to ensure that all the different components in a camera or lidar system and there are many are all working optimally and aligned precisely breaklines best practices are things like consistent digitizing of break lines complete digitizing of features as brake lines etc um the current methods are numerous every project will have a different specifications for spacing and accuracy it’s important that the provider and the client understand what specifically these specifications are and how to best design the acquisition and the delivery of the products to meet all those specifications next slide please and with that we’re we’re finished with the presentation and are happy to address any questions and I’m going to

turn this over to Josh to lead us through that phase thanks Mike and Chuck great presentation very informative and very useful information so I hope our audience shares that Center thanks for sharing your knowledge with us so now I’ve got questions for our attendees if you already have some coming in appreciate those we do want to note that we want to stay away from being overly project specific with the Q&A so if you have any product specific type questions you can feel free to add them here but we will have to get back to you after the event on those because I think the relationship but generalized questions feel free to add them now you can also feel free to call us anytime hastens of that ends here era services or with sis and discuss lidar ortho accuracy issues even if it’s just general questions and you just want to vet those so that even if you don’t have specific projects that you are going to request with us so with that again so for dad questions as we go on here and I’m going to ask one that came in from Adam to have both of you addresses of you if you’re so willing the question is with regards to the pixel mismatch that was discussed earlier Partho photos is that applicable only to the ground or clinic also applied to a brown object like transmission line catenaries and other above-ground features how accuracy apply to that specifically with regards to orthos the little part of that question can you repeat that one more time sure the question was regards to pixel mismatch and orthos is that doesn’t apply to the ground only or can it also apply to a brother objects like transmission lines etc oh sure yeah um absolutely a pixel mismatch can apply to any feature in the ortho whether it’s the ground or a tower or a building or bridge so yes it can apply to any feature visible and the oracles let me let me add to that if he’s referring to the next-generation specification for the sceneline mismatch in in the pixel for the different next-generation classes one two and three a B and X then I’m not certain how they’re going to apply that but what that stated is is the maximum mismatch along along the seam line so if the seam line for the ortho data cuts through our truncates a feature that’s above ground I have to think that that they that would be part of the error criteria and looking at how well they match there may be some leeway in whether it’s a road feature or an above ground feature but I think that that’s part of these new standards that’s going to have to be ironed out yeah he was addressing the seam line mismatch after this that’s what he was asking so thanks thanks they’re talking oh sorry Mike well I just kind of just going to add to that so if if he was talking about that mosaic seam line mismatch then yeah the specifications will say that a mismatch a mismatch of some proportion is acceptable 1 2 pixels but beyond that the feature should not mismatch and it can’t apply them any feature whether it’s on the ground or not great all right so Deena a question with regards to yes are there big different big differences between a STRs and as da standards and also I would threaten the giraffe specifications with a STRs that we showed well it look like the slides that we noted are that Chuck went through with the different levels of actually that is that the actual draft or definitely we came up why don’t you want to take that sure that that is an actual copy of the draft I want to say it’s like version six of the draft so yes those tables were actually extracted from that the draft digital geospatial data accuracy standards and we’re going to try to provide but anyways I’m sorry I was just going to say that we were going to try to provide some of that information and

links and whatnot in the post-event emails to all of you that are attending and registered so you have a additional I’m trying to look over all that so the first part of that question Josh was that how there’s how do the ASPRS versus the NS SDA specifications very was yes yeah one of the big differences are there big differences between those two standards well ASPRS standards are typically based on the maximum rmse in in X Y or Z and they would tie that to either a scale as well as in the vertical you know a contour requirement such that that the RMS E&Z for a particular contour interval would have to be one third of that interval so the the ASPRS stay energy that we see are tied to scale and they are related to our MSE whereas NSS da is computed from an our MSC product in a 95% confidence multiplier Z value thanks Chuck we are just a note to our Chinese where are at three o’clock which was the original end time for this event we do have a number of additional questions so I think Mike and Chuck can hang on the line and answer those so we’ll just continue on if you do have additional questions go ahead and throw them out now so we can try to get them in before we end the session so let’s go on Robert had a question I’m going to read this word for word that the you had a very specific question when the distribution of accuracy check measurements don’t approximate to the standard bell curve the mathematical relationship constants between the rmsd and 95% confidence don’t seem to hold true can you comment on this and have how you have you had any experience with this does that ring a bell with you Chuck or Mike quick haha sure when when let’s let’s say that we were looking at at something where the errors are not either normally distributed or where the distribution is not circular and where the Delta is in X or Y predominate in those cases the error distribution is more elliptical and and in those cases the RMS e and one or another direction may predominate so you know if we if one if we don’t have a sufficient number of check points or two if the errors are not normally distributed we may see that the data either are grouped if it’s if it’s systematic bias within certain areas in the circular air plot or they all point to it in a certain systematic direction and the vector offset plot or there just aren’t sufficient points in order to get a data set so we would see values that are nonzero mean rmse in X or Y that may predominate or bias statistics that would give us indication that we don’t have either sufficient check points or that the data have some kind of bias and I might just add that we do see this permit does occur these statistical measurements of air do make assumptions about a normal distribution of air around a mean that they do assume that there’s circular air and not elliptical air that the air and X is about the same as the air in Y for example circular so but we do see cases where there is elliptical air the air and X is more than the air and Y for example we do see datasets where the air is not normally distributed in those cases when we’re using proper qa/qc tools and methods we will observe that what will be tected we can measure it we can we can quantify this non-random randomness these non circular errors and then we can construct an approach to deal with it we can either reinvest how we collected control or measured checkpoints maybe we didn’t use random randomly located

points as checkpoints maybe we need to add more control points to the data set to eliminate some of the non circular or non random errors so or sometimes we because of the project constraints there’s nothing we can do except to report it and indicate in the metadata to the client that this is what the result is and here are the options I’m going to add to that really quickly Robert when when when we have a situation where it is non circular the F the the fgdc or NSS da or ASPRS guidelines I’ll do point back to Greenwald and Shultz to some extent but they’re all simplified into two special one in which our MSE X&Y are equal that’s case one and in case two is when our MSE and X does not necessarily equal our MSE my and they give two simplified equations but if you go back to the original greenwald insults diagram they have a series of values where they take this actually standard deviation in X&Y and they compute ratios of the minimum to the maximum and then they they look at the relative values in those and then they have linear combinations that are used to compute the error statistics based on the relative values of those what we see in practice though is that the fgdc n SS da guidelines have simplified that so there are a lot of assumptions and how those are applied but if you really want to dig in to the governing accuracy equations you can always go back to Greenwald and Schultz and look at how it would be more rigorously quantified but in general practice from professionally acquired data using best practices and methods it’s usually safe to use the governing equations and the simplified methods that are applied in n SS da and ASPRS specifications and standards thank you guys very good answer thought out thank you for trying out for Robert okay so the next question I have in the list here is someone was asking specifically a housekeeping item on will this presentation be available via audio after the conclusion I’ll take that since I’m the one that will be preparing that yes we will provide this session via video and audio is the intention via their services website and all of our attendees and registrants of this will receive a link to that will also attempt to provide slides and the some other links and references and some of these certifications that we’ve talked about during this session today so look for that and hopefully as long as all of our post treasure goes well that’ll happen soon next question was the guards too oblique imagery dan was asking what work has been done for testing accuracy of oblique type imagery 45-degree angle imagery that we’ve seen become popular over recent years Mike I know we’ve seen mixed results with that Chuck I’m not sure if you guys have ever done and you work with regards to that country specifically I don’t know is the short answer I don’t know how much rigorous research has been done to to quantify air and oblique imagery it’s a good question it’s a whole different type of product because the scale is constantly variable from the center of an oblique out to any edge the scale is different so the air is different so Chuck have you got any experience with obliques I only on the research side we’ve looked at some of the the tools and the methods of measuring elevations using oblique imagery and it as you say the feet depending on the feature of the scale and the location it can widely vary we we have not put a lot of effort into creating tools that automate those processes associated with the oblique there are ortho deliverables that we’ve looked at and and we’ve done some testing of but you know nothing to do in depth because we really haven’t had a data set that’s been compiled to me or that it’s been delivered to us to say use this using check this using check point analysis and verify the horizontal

or positional accuracy so there’s there’s really not been that much on our side of verification of oblique if there’s anybody listening that the knows of statistical methods or even standards for air in oblique photography send them to Josh at the email address he’s what I’d like to know it I’m not sure that there are any standards for oblique imagery again because it’s a different animal it’s a it’s an image unlike an Oracle that has constant scale a fixed scale an oblique has continuously variable scale and therefore the air at any point in the photograph is different than the air associated with another point somewhere else on the program okay thanks guys looks like we have one more question for sure here before we wrap up how do you generate brake lines vertical imagery lidar Siri models are 2d digitizing and the draping the wire over that data set Mike I know we’ve got experience with this do you want to start off with that sure um basically there’s several methods the best and most preferred method is to do it from stereo imagery so with sterile imagery you can you can see the features in the photographs in the in three dimensions and you can very accurately see the bottom of a ditch or the top of a feature and digitize it in V a second method is is to use lidar geometry where you have your lidar points available to you in stereo and canned in and detect where the bottom of something is from the lidar points themselves or the top of the feature and digitize that way a third method is to digitize feature from heads-up digitizing or from a 2d photograph that’s not as accurate you don’t have any 3d information so you’re digitizing the feature from and they’re laying that line over the top of another surface from a dam to give it a third dimension so stereo photos are the first and best method of digitizing break lines well great if there are no other questions I think we’re gonna wrap it up here appreciate the talk Chuck and Mike it’s been great appreciated all the questions being answered on behalf of our attendees and again to our attendees please note that we will be providing links following the event here within the next week with audio video slides and other references to the view what was discussed here so thank you all for coming don’t forget to fill out the survey as you leave it will appear and thank you again we’ll help with those see you again soon perhaps at next month’s UAV and geospatial webinar August 22nd thanks everyone thanks guys Thank You Joshua thanks Mike