Evaluation Methodology for the Participatory Evaluation of the Pehuen Foundation, Alto Bio Bio, Chile.
Fall 1995
Theodore E. Downing, University of
Arizona
Scoping and Design
1. The evaluation was conducted
in two phases. In the scoping and fieldwork design phase, the evaluator,
Theodore E. Downing, reviewed relevant background information and
documentation on the Foundation at the IFC offices in Washington, D.C. Documents and correspondence after 15
October were not made available to the evaluator. Based on this information, a
draft analytical framework was prepared for evaluating the Foundation based on
areas of analysis associated with each of its four statutory and/or legal agreement
objectives (Table A1.1). Following an IFC and Pangue review of the draft framework,
extensive consultations began in Chile with stakeholders who had been
identified by Pangue, the IFC, or discovered in the document review (17 Sept -
5 October 1995). The evaluator also made a brief site visit to the Alto Bio
Bio and attended a Foundation board
meeting and an assembly in Callaqui.
2. At the time of the review of
the analytical framework, the stakeholders were informed about the intended
scope of the interim evaluation to avoid any misunderstanding that this was a
special review. Many made methodological
suggestions and all exposed the evaluator to the diversity of public perceptions
about the Foundation. The framework was
reviewed and modified following discussions with over 61 stakeholders in the
public sector, NGOs, environmental groups, academics, consultants, the
Foundation Board and staff located in
Santiago, Concepcion, Temuco, Alto Bio Bio and the United States and leaders
from Callaqui, Pitril and Quepuca-Ralco. In some cases, stakeholders proposed
specific indicators or measures be examined to determine Foundation
progress.
3. There was broad based
support for the Foundation and IFCs general objectives and all consulted
unanimously agreed that the proposed analytical framework for evaluating the
Foundation was acceptable and comprehensive.
A few stakeholders dismissed the interim evaluation as too narrowly
constituted, feeling that the evaluator should focus on the underlying motives
of Pangue and its Foundation as an instrument of penetration of Endesa and its
subsidiaries into the Alto BioBio.
Since this was not a statutory or an Agreement objective, the evaluator
explained that this issue fell outside the terms of reference.
4. In many consultations, a
hypothetical example of the full
development of a methodology for an area of analysis was provided in each
consultation. Stakeholders were given blank forms and given an opportunity to provide methodological
suggestions. Given the technical nature
of this request, they were permitted to complete the forms after the meeting
and FAX them back to Santiago by October 2. On 2 October, a secretary in
Santiago made follow-up calls, encouraged replies, extended the deadline
another three days. The stakeholders were also requested to suggest methodology
approaches appropriate to Pehuenche culture and identify perceived strengths
and weaknesses of the Foundation which might be examined. Only a handful
responded. Both the field methodology and framework were approved by the IFC on
11 October 1995. The Foundation
President, and Pangue S.A. also accepted it as a comprehensive approach to the
evaluation.
5.
During
this phase, the evaluator added a volunteer indigenous affairs specialist,
Carmen Garcia-Downing from the World Health Organization Collaborative Center
at the University of Arizona, a Zapotec Indian from Southern Mexico with
extensive experience in working with rural indigenous peoples. She also
participated in the field evaluation phase.
Field methodologies
6. The
field challenge was to
develop a simple, participatory evaluation combining conventional evaluation
techniques and participatory methodologies which will be capable of detecting
short-term changes in socio-economic and cultural status. The initial field
site survey revealed that the Foundation and other government agencies were
having difficulty communicating with the Pehuenche. Although many younger
Pehuenche are bilingual, some of those over thirty have only marginal command
of Spanish and the CEDEP survey revealed that Chedungun is spoken in 61 percent
of the households, and, in 23 percent, it was the only language. The evaluator also discovered that the
people had never been actively involved in an investigation of themselves, even
as collaborative field assistants. Ethnographers, environmentalists, critics of
the hydroelectric projects, government agencies, and academics had consistently
treated the Pehuenche as objects of study rather than active participants.
7. To mitigate the logistic and
cross-cultural communication difficulties and increase the accuracy of the
work, the evaluator used a multiple method evaluation approach, mixing a
battery of participatory and conventional evaluation techniques and advanced
ethnographic methods (Table A1.2).
|
|
|
Field group meeting tent |
8. Logistically, a basic
problem was where to meet in order to maintain the independence of the review.
The Pehuenche communities lack a central, community meeting place. The Foundation had recently built small Foundation headquarters buildings in
Pitril and Quepuca-Ralco, but meeting there might have compromised the indepedence
of the evaluation. During the scoping
phase, the Pehuenche Board representatives cautioned the evaluator that walking
to the government service centers in each community was an imposition. The
evaluator witnessed this problem first hand in Callaqui, when a lonko assembled over forty residents for
a meeting in a freezing rain. On advice of the Foundation Pehuenche representatives
and as a courtesy to the Pehuenche, the evaluator held most of the 11 focus
groups in a large tent placed at sites designated by the Pehuenche leaders and
representatives to the Foundation. The
tent proved a useful methodological innovation and Pehuenche sectors requested they be permitted to use it for small
meetings. At the end of the field session,
it was donated to the Foundation with explicit instructions that it is for the
exclusive use of Pehuenche sector meetings, upon their request.
9. Only three or four Pehuenche
in the Alto Bio Bio were known to have a high school education. To conduct a participatory survey within the
time constraints, the independent evaluator and the indigenous affairs
specialist hired and trained a 6 person, Pehuenche evaluation team (two from each
community, 3 men and 3 women). A
seventh Pehuenche research assistant from Trapa-Trapa joined the team in
midstream and completed specialized assignments. The team was trained at a rudimentary level to interview,
administer questionnaires, assist in participatory voting, translate, record
focus groups, and conduct specialized tests.
The team demonstrated considerable initiative and provided high quality,
reliable data and the community took pride in their participation. [ted1]
10. The advantages of using the
local Pehuenche team far exceeded its drawbacks and the delays it caused. Initially, they were uncertain whether or
not they could perform the work, but their confidence rapidly increased to such
an extent that they requested more formal training so they might form an
evaluation team. All were highly
motivated and energetic and their horsemanship skills proved most useful as the
random sampling seemed to always select the most remote households. The
reliability and accuracy of the data was substantially increased. They had extensive knowledge of their
communities, were trusted by their own peoples, had personally participated in
Foundation programs, and understood Chedungun. The disadvantages were minor and
mostly related to the minimal education and inexperience with machines. The
training period delayed completion of the evaluation, all data had to be
carefully reviewed. Their data entry capabilities were just beginning to show
promise at the end of the field session. Most significantly, each community
meeting brought positive feedback for the team from the community since this
was the first time that Pehuenche had conducted research on their own community.
11. Over a seven week period,
the team completed 11 focus groups, ranging in size from 5 to 24 people with an
average of 15 people per meeting.
Meetings were held in different sectors of the community to permit
higher attendance and minimize the imposition caused by the evaluation. The
gender ratio was 60:40 ratio of males to females. A separate meeting was held
with non-Indian inhabitants of Pitril, at their request. The indigenous affairs
specialist held an all female focus group of 22 women in Callaqui Alto. Women
only meetings are valuable, but unusual in Callaqui, and, within hours, some
Callaqui males tried to discredit the focus group, but were unable to make
reference to any specifics.
12. Pehuenche research
assistants were trained to administer
resource allocation priority test to a random sample of 60 households (Callaqui
22, Pitril 14, and Quepuca-Ralco 24). The entire Pehuenche research evaluation
team administered a community resource allocation priority test in each
community at the close of the field session in which 183 people participated
(Callaqui 57, Pitril 71, and Quepuca-Ralco, 55). This permitted the evaluator to determine if Pehuenche voted
differently in groups than as individuals.
13.The Foundation staff
graciously received the evaluation team and granted it full access to
Foundation records. The new Executive Director was extremely helpful, but
unaware of many aspects of the program. The Foundation has no on-going
monitoring function and the diagnostic surveys were not being used, apart from
providing an incomplete list of family numbers and names. On request, the Executive Director and staff
culled their files and organized their first complete list of all community
members who had obtained benefits from 1992-95. It is hoped that this list
should provide the Executive Director
within invaluable information for monitoring the DGP and resolving conflicts in
the field. The list was finished on the
last week of the field session. The
result was a 6 week delay while the Foundation office records, CEDEP survey
records, and the evaluation team survey data were collated.
Participatory tests
12. A battery of participatory
instruments, some proprietary to Downing and Associates, were used (Table
A1.3). They include:
13. Photographic theme analysis. A random selection of families from the focus group participants were
given disposable 35mm cameras. The
investigators taught the randomly selected person or another member of the family
on how to use the simple camera. The new photographers were asked to take photographs of 1) things of
family or community importance, 2)
family, 3) favorite purchases from the Foundation and 4) dream projects (i.e.
personal or community works) that they
would like to see carried out. The
cameras were later collected for film development. Once the pictures were
developed, the Pehuenche evaluation assistants went over each photograph with
the photographer and tape recorded the meaning the photographer wanted to
convey in the picture.
14. Marginality Pile Sort Test (MPST). The marginality
pile sort test provides an indicator of the relative wealth of households
within a community. In each community,
Pehuenche field assistants were trained to administer a marginality pile sort
test. A random selection of 36 households
was drawn (Callaqui 14, Quepuca-Ralco 10, Pitril 12). A set of cards were made, each listing the name of a head of
household. The Pehuenche research assistant read each name to the respondent,
asking them to place the card in one of three piles according to their relative
wealth within the community. Once completed, the assistant coded the ranking on
the reverse side of the card. For each household, their average rank was
calculated. Since people were unfamiliar with the economic station of people in
other communities, the MPST had to be community specific. Inter-respondent consistency
tests were used to identify statistical divergence. The ranks were then entered
into the master data base created for the population which included the value
of all contributions obtained from the Foundation from 1992-95, CEDEP data, and
additional information scored for each household by the Pehuenche research team
(e.g. migration, source of income for the family, and so on).
15. Discount Group Purchasing
Program (DGPP) participation, sources
of family financing for each project and saliency test. A random sample of 41 individuals for the three communities (with the exception of two households in
Callaqui Bajo) were interviewed. Pehuenche research assistants, working in 2
person teams, asked interviewees to name all group purchases that they made
from the Foundation (called proyectos).
People were also asked which projects they liked most, least, whether or not
they had organized a purchasing group,
and what was their source of financing for the purchases. If they had
not participated in the Foundation DGPP, the reasons were requested. The
frequency of shared responses and order of response was used to measure saliency
of purchases and source of income. Interviews were conducted in Spanish or
Chedungun, as desired by the interviewee.
16. Cognitive proximity tests. A test developed by Downing and Associates to determine how
people perceive changes in their institutional environment, the evaluator used
a cognitive proximity or triad test.
Direct interviews determine the elements of a semantic domain (for
example, in English, dog, wolf ,cat , horse are all part of what might
be called the domain mammals). Within
that domain, some elements are closer (more proximate in technical jargon) than
others, e.g. in a triad of dog, wolf, cat, most would say that cat is least
alike the others. Two triad tests were
administered, one on the domain of institutions (in Spanish/Chedungun) and the
other on types of material exchange exclusively administered and answered in Chedungun (data used in para. 51-52 and
Figure 1).
17. Resource Allocation Priority
Test (individual and community versions). This test was designed by Downing and Associates
to determine the relative importance of different program options. For this
test, a person was given 5 marbles, each representing 10
million pesos (roughly the annual Foundation direct expenditures for the 3
communities). The persons were asked to
indicate their preference for allocating this money among three alternatives.
The alternatives were derived from the focus group discussions and
direct interviews. After the priorities
were clearly discussed, each respondent distributes his or her secret votes (marbles) into a cloth with three
pockets. Explanation in Chedungun was given
by one of the Pehuenche assistants throughout the discussion, and while
the person stood in line waiting to vote, another Pehuenche research assistant
provided additional explanation of the procedure and meaning of the vote to
each person. An odd number of marbles
is used to force allocation priorities. When the test is administered to
individuals, the pocket chart is unnecessary (data used in para. 241-246).
|
|
|
Voting during the intracultural communications test |
18. Inter-cultural communication
test. A
young Pehuenche artist prepared five diagrams of different forms of interpersonal
communications between two men - one, a non-Indian representing the Foundation
(huinca), and the other a Pehuenche
(Figure A1.1). In 8 focus group
meetings, the pictures were laid on the ground and used to initiate a
discussion about the ways people communicate with one another. Attendees were given twigs and asked to lay
a twig on the painting that most closely represents a) how the Foundation is presently communicating
with the people and b) how they would like to interact with the Foundation. The
votes were tabulated and the results discussed. Before each vote, a Pehuenche field assistant described the test
again in Chedungun. People were very
interested in this test. The same test
was administered to the Foundation staff.
19. Reanalysis of other survey
data. The
evaluation took advantage of previous surveys. Systematic census data was
available from the Foundation sponsored CEDEP household survey of 1993 from
household regular interviews and from surveys by municipal welfare agents. Critical,
previously unavailable time series data (running records) were counted to detect changes in resource
utilization. This included a laborious count of over 4000 transportation
permits for timber or animals.
20. Conventional evaluation
methodologies.
Conventional methodologies also included workshops with the Foundation Board
and staff. The Provincial Governor organized a intensive review of the
Foundations regional impact. The workshop included participation from all
government agencies and an NGO working in the region.
21. A serious methodological and
analytical problem emerged during the evaluation as it became apparent that
each community had a distinct profile, much of which was a result of
differential contact with non-Indians and different access to natural
resources. Moreover, factionalism had
reached such a level in Callaqui and parts of Quepuca-Ralco that focus groups
with different faction members were impossible. Individuals refused to attend
meetings if the other faction was present. In one case the evaluator was warned
of the potential for physical violence if we combined the factions in a single
meeting. This required samples to be
drawn for each community rather than all three together, which tripled the work.
Evaluation team
23. Team leader: Theo. E. Downing, Social Development. Carmen Garcia-Downing, Indigenous Affairs Specialist. Pehuenche Evaluation Team: Team Supervisor: Luis Vita Vita, Member: Lucrecia Levi Sandoval, Lucenda Levi Levi, Norma Beltran Manquemilla, Jose Feliciano Purran Manquemilla, Segundo Vita Carasco (videographic assistant). Special assistant, Jacinto Manquepi Vivanco.
24. There is extensive interest
in the Pehuen Foundation throughout the public and private sector of Chile. The
evaluator is extremely grateful for the time and energy given to provide a
complete perspective. While listing
their names and affiliations may be the most efficient form of acknowledgment,
it certainly does not reflect the quality of the interaction and the depth of
my gratitude. The list is incomplete
since it does not include the hundreds of community members who walked many
hours to meet us and openly expressed their concerns for the Foundation and the
role it might play in alleviating them from their suffering.
Pehuen Foundation
Participatory Interim Evaluation Team, 1995
|
|
Table A1.1 Analytical Framework
Foundation Objectives |
Area of analysis |
|
A.
Poverty reduction. Improvement of the socio-economic conditions of the Pehuenche
communities of Callaqui, Pitril, y Quepuca Ralco in a sustainable manner |
Reduction
of marginality a) housing b) health c) education d) income |
|
B.
Cultural identity. Promote and reinforce Pehuenche culture |
1.
Pehuenche
self-identity 2.
Ownership of the Foundation by the Pehuenches |
|
* C.
Sustainable development. Provide a sustainable development which will provide
long-term benefits to the Pehuenches |
1.
Sustainability of the Foundation, as an organization a.
governance b.
management c.
fiscal
affairs d.
program
design and development e.
external
relations 2.
Long
term development of resources a.
Pehuenche
human resources b.
community
and family physical infrastructure c.
renewable
natural resources d.
community
organizational capacity |
|
* D.
Impact management. Mitigate the potential post-boom impacts following construction
activities of the Pangue hydroelectric project |
Impact risks
commonly associated with unmanaged impacts a) health
risks b) landlessness c) marginalization d) homelessness e) socio-cultural disintegration f) loss
of food security g) loss
of access to common resources h) loss
of income |
* IFC and Pangue jointly agreed upon this objective which is not part of
the Foundations statutes
Table A1.2 Major
methodological decisions
|
Unit of study |
Entire
populations of Callaqui, Pitril and Quepuca-Ralco, including people who are not
members (socios) of the Foundation |
Cultural considerations
|
1. Extended
period in the community 2. Identified
and trained local Pehuenche as research assistants 3. Adaptation
of field techniques for illiterate or minimal literate population 4. Use
of translators 5. Distinct
focus groups: men and women together, women only 6. The
energy and time which Pehuenche gave to the focus group and community-wide
polling participants was rewarded with raffled tickets for a lottery at the
end of the field session. This method was judged culturally appropriate and
exciting by the Pehuenche evaluation team. 7. Chedungun-Spanish
bilingual to assist in focus groups |
Confidentiality and independence
|
1. Names
of interviewees were coded and remain confidential 2. Held
interviews inside a tent rather than holding them at the
Foundations local storage room. This was to create an atmosphere of
neutrality so as to allow non-members of the Foundation to express themselves
Lonkos suggested strategic meeting
points so as to minimize participants travel time to the focus sessions |
Field period
|
1. Oct
- December field session (before
transhumance to summer pasture begins) |
|
Evaluation Methodologies |
|||
|
|
Participatory |
Conventional |
Advanced
ethnographic |
|
A. Reduction of poverty |
1. Community focus groups 2. Photographic theme analysis 3. Individual, directed interviews 4. Discussion groups with service providers (e.g. health, education, agriculture and livestock extension and municipal officials |
1. CAS II Poverty indicators 2. Select Running records 3. Selected quantitative indicators in areas of concern, extrapolation from key indicators 4. Further analysis of Pehuen Foundation survey data 5. Direct observation (using guide) 6. Analysis of Pehuenche participation in the DGPP from Foundation files |
Marginality Pile Sort Test |
|
B. Cultural identity |
1. Community focus groups 2. Individual, directed interviews 3. Group interviews with Pehuenche leaders 4. Photographic theme analysis |
1. Directed observation 2. In-depth interviews 3. Analysis of Foundation purchases |
1. Cognitive proximity tests in Chedungun 2. Intercultural communication test |
|
C1 Sustainability of the Foundation |
1. Focus group with Pehuenche Board Members only 2. Reflection group w/Board of Directors 3. Reflection group w/Foundation Staff 4. Group interviews with Pehuenche leaders 5. Workshop hosted by the Provincial Governor |
1. Reanalysis of CEDEP survey 2. Analysis of Pehuen methodology, including delivery mechanisms and outreach 3. Review of Pehuen Foundation reports, minutes, consultant reports 4. Case studies of projects Comparison to NCIB standards |
Cognitive
proximity test |
|
C2. Sustainable, long term development of
resources |
1. Community focus groups 2. Reflection group with Pehuenche leaders 3. Group interviews with Pehuenche leaders 4. Reflection groups w/Board of Directors and w/Foundation Staff 5. Discussion groups with service providers (e.g. health, education, agriculture and livestock extension and municipal officials |
1. Selected quantitative indicators in areas of concern, extrapolation from key indicators 2. Reanalysis of other survey data (running records) |
1. Discount Group Purchasing Program (DGPP) participation, sources of family financing for each project and saliency test 2. Resource allocation priority test 3. Voting with marbles |
C2.
Sustainable, long term development of resources
|
1.
Community focus groups
2.
Voting with marbles
3.
Reflection group with Pehuenche leaders
4.
Group interviews with Pehuenche leaders
5.
Reflection groups w/Board of Directors and w/Foundation Staff
6.
Discussion groups with service providers (e.g. health, education,
agriculture and livestock extension and municipal officials
|
1.
Selected quantitative indicators in areas of concern, extrapolation from key
indicators
2.
Running records
|
1. Free listing using a snowball sample
|