Implementing Impact Assessment
Benchmarking
Benchmarking uses information of similar organizations or processes to assess the gap between current and achievable performance. In the context of agriculture, benchmarking usually refers to a systematic comparison of the performance of farms producing under similar bio-geophysical conditions.
Benchmarking assesses performance in relative, rather than in absolute terms. This is particularly suited for assessing effects of agricultural management, because the strong influence of external factors such as climate, weather or soils on farm performance is difficult to isolate. Simple benchmarking identifies best practices from a set of samples and then evaluates all samples relative to these best practice examples. In more complex benchmarking where multiple evaluation criteria are applied, samples are not evaluated relative to a single best practice but relative to a so called efficiency frontier, which represents the best performances considered achievable and which is calculated based on data from all samples. The most common parametric and non-parametric approaches to this are Stochastic Frontier Analysis (SFA) and Data Envelopment Analysis (DEA) respectively.
Benchmarking is a relative performance evaluation that compares the efficiency of similar organizations or processes. It is used in many sectors. In the context of agriculture, benchmarking usually refers to the systematic comparison of the performance of farms that produce under similar bio-geophysical conditions, using the same resources and producing the same type of products.
In the context of benchmarking, efficiency is defined more loosely than in the definition of resource use efficiency described in the previous sections. Here, efficiency implies a maximization of positive outputs and a minimization of inputs and of negative outputs. Evaluations of efficiency can be based on very different criteria, such as on provision of ecosystem services, effects on biodiversity or resource use. All impact areas presented on the BonaRes Assessment Platform can therefore be used in benchmarking.
Simple benchmarking first identifies best practices within a set of samples. It then evaluates all samples by assessing the difference between their performance and the performance of the best practices (Oh & Hildreth, 2016). More complex benchmarking methods that apply multiple evaluation criteria use data from all samples to calculate a so called efficiency frontier. This frontier represents the best performances considered achievable. The performance of all samples is then assessed relative to this frontier. The most common methods for this are Stochastic Frontier Analysis (SFA) and Data Envelopment Analysis (DEA).
Benchmarking is often conducted to improve organizational processes and to achieve higher standards of performance. This can be accomplished by learning from the experience of best practices or by copying their methods or processes (Malano et al., 2004). Benchmarking studies can also be used to assess the impact of policies (Bogetoft & Otto, 2010).
The performance of farms strongly depends on bio-geophysical site conditions. Where assessment seeks to evaluate different types of soil management, it is challenging to isolate and quantify the influence of external factors. For example, if a farmer in Finland achieves a harvest of 4.5 tons of wheat per hectare and a farmer in France achieves a yield of 7.0 tons, how much of this difference is then due to their respective managements and how much is an effect of the different agro-climatic zones?
To avoid such problems, performance is often measured in relative terms through benchmarking. In agriculture, benchmarking plays an important role in the context of assessing environmental impacts of farm performance, such as impacts related to water conservation or soil protection (Kuo et al., 2014). Results of benchmarking studies are useful at multiple decision making levels, enabling policymakers to assess the effect of agricultural policies (Quiroga et al., 2017) and allowing farmers to learn from best practice examples.
Benchmarking studies can be applied with different degrees of complexity. In the simplest form, only one criterion is used to evaluate performance and a best practice is identified which scores highest for this criterion. The best practice is then used to evaluate all other samples and to serve as an example. For instance, several wheat producing farms in a region could be compared in a benchmarking to identify the one that achieves highest yields. All other farms could then be evaluated relative to this and farmers could try to use the example to optimize their own management.
In a more complex form, several criteria are evaluated and the information of fall samples are mathematically combined to form a so called “efficiency frontier”. The most common parametric method for this is Stochastic Frontier Analysis (SFA), the most common non-parametric method is Data Envelopment Analysis (DEA). Both methods have been developed as economic tools to evaluate the performance of companies, enterprises or non-profit organizations, which are referred to as “decision-making units (DMU)” in the literature (Malana & Malano, 2006). The abstract term DMU is used to highlight that the methods are applicable for all types of organizations where management (decision making) occurs. In the context of soil related impact assessments, the DMUs are usually farms.
Both SFA and DEA address the question of how the highest amount of positive outputs can be achieved with the lowest amount of inputs and the lowest amount of negative outputs . For this, they calculate an efficiency frontier line, which represent the (hypothetical) optimal combination of inputs (Oh & Hildreth, 2016). The efficiency of each DMU is calculated as the distance to this efficiency frontier and often reported as a percentage (with DMU positioned on the frontier line having an efficiency of 100%). SFA and DEA can be used to estimate overall efficiency of farms or management processes and rank them based on their performance. Additionally, the methods can be used to identify areas of inefficiency in order to support improvement.
Stochastic Frontier Analysis (SFA)
Stochastic Frontier Analysis is a method developed from stochastic frontier production function models (Aigner et al., 1977). SFA is based on parametric models. Such models are defined a-priori except for a finite set of unknown parameters that need to be estimated from data. In SFA, these parameters often include effects of noise (random shocks unrelated to management, such as weather effects) and efficiencies (Bogetoft & Otto, 2010). An efficiency frontier is calculated based on actual input/output relationships, and the efficiencies of all DMU are measured relative to it. It is not necessary for any DMU in the sample to be positioned on the efficiency frontier, i.e. to achieve full efficiency.
With SFA approaches, hypotheses about production structure and degree of inefficiency can be assessed statistically. However, the method requires explicit parametric functions representing relationships between inputs and outputs, and distributional assumptions. For considering measurement errors and effects which cannot be controlled by the individual DMU, such as weather, an error term is incorporated in the calculations. The validity of SFA results depends on the accuracy of the assumptions made (Odeck, 2007).
Data Envelopment Analysis - DEA
DEA calculates the efficiency frontier line (highest positive outputs from lowest amount of inputs and negative outputs) from linear combinations of the samples. DMU that score highest in at least one criterion will always be positioned on the efficiency frontier. Two types of DEA models can be distinguished: The input-oriented model aims towards minimizing inputs while holding the amount of outputs constant. Output-oriented models on the other hand aim to increase outputs with the same amount of inputs (Malana & Malano, 2006).
As a difference to parametric models, non-parametric approaches such as DEA do not require any a-priori assumptions regarding the quality or distribution of the data, avoiding potential errors in this regard (Oh & Hildreth, 2016). However, this makes the methods sensitive to measurement errors or other noise affecting the dataset. In DEA, all parameters are calculated from the data in the sample. Environmental factors are not explicitly considered. It is therefore important to carefully select the sample to avoid comparing farms that work under different environmental conditions (Odeck, 2007).
Both SFA and DEA allow to evaluate performance with regard to multiple inputs and outputs and facilitate an integrated evaluation (Bogetoft & Otto, 2010). Both SFA and DEA also have specific advantages and disadvantages to be considered when choosing methods. Alternatively, the methods can be combined to overcome the disadvantages, and such combinations have been applied in several studies dealing with benchmarking processes (Oh & Hildreth, 2016).
Aigner D; Lovell C A K, Schmidt P. 1977. Formulation and estimation of stochastic frontier production function models. Journal of Econometrics 6 (1), 21–¬37. DOI: 10.1016/0304-4076(77)90052-5.
Bogetoft P, Otto L. 2011. Benchmarking with DEA, SFA, and R. International Series in Operations Research & Management Science, Springer, New York, U.S.A, 352 pp. DOI:10.1007/978-1-4419-7961-2
Kuo H, Chen H, Tsou K. 2014. Analysis of farming environmental efficiency using a DEA model with undesirable outputs. APCBEE Procedia 10, 154–¬158. DOI: 10.1016/j.apcbee.2014.10.034
Malana N M, Malano H M. 2006. Benchmarking productive efficiency of selected wheat areas in Pakistan and India using Data Envelopment Analysis. Irrigation and Drainage 55, 383–394. DOI:10.1002/ird.264
Malano H, Burton M, Makin I. 2004. Benchmarking performance in the irrigation and drainage sector: a tool for change. Irrigation and Drainage 53, 119–¬133. DOI:10.1002/ird.126
Odeck, J. 2007. Measuring technical efficiency and productivity growth: a comparison of SFA and DEA on Norwegian grain production data. Applied Economics 39: 2617–¬2630. DOI:10.1080/00036840600722224
Oh SC, Hildreth A J. 2016. Analytics for smart energy management. Springer Series in Advanced Manufacturing. Springer International Publishing, 295 pp. DOI:10.1007/978-3-319-32729-7
Quiroga S., Suárez C., Fernández-Haddad Z., Philippidis G. 2017. Levelling the playing field for European Union Agriculture: Does the Common Agricultural Policy impact homogeneously on farm productivity and efficiency? Land Use Policy 68, 179–¬188. DOI:10.1016/j.landusepol.2017.07.057