Using Choice-Based Conjoint Data as Input in ELECTRE II: University Preference Case in Turkey

The main objective of this study is presenting usage of conjoint data in one of the multi criteria decision techniques (MCDM) ELECTRE II in the context of decision-making process. The approach has been implemented by establishing an objective ranking of private universities in the context of university candidates’ preferences. ELECTRE II procedure is performed on the factors affecting the preference of private universities among candidates and the investment expenditure distribution of the universities. Preference data were collected by Choice-Based Conjoint (CBC) method from 296 students who were in the preference process after 2016 university entrance exams in Turkey. The results obtained from CBC were used as input in ELECTRE II in order to determine a complete and objective ranking of universities. As a result, it could be seen how the rankings differ according to student preferences when investment expenditure areas of universities change the weights of factors. This approach also allowed us to describe the market situation in general thus each university could make a comparative assessment of its own.


Introduction
Conjoint analysis (CA) has a widespread usage in determination of consumer preferences with its different approaches after it was developed in early '60s [1,2]. A well-known approach in conjoint measurement is called Choice-Based Conjoint (CBC) and it revealed strong acceptance in marketing research after McFadden's 1986 study [3]. Lately, conjoint scores started to be used as an input for Multi-Dimensional Decision Making (MCDM) methods which run a ranking procedure, such as ELECTRE (Elimination Et (and) Choice Translating Reality) [3]. The technique has six different variations, namely ELECTRE I, ELECTRE II, ELECTRE III, ELECTRE IV, ELECTRE IS and ELECTRE TRI (B-C-nC). ELECTRE II was developed by scholars Roy and Bertier [4] as a MCDM technique that provides rankings and superiorities of different alternatives according to their attributes' performance scores. Evaluation method of the technique is based on pairwise comparison of alternatives by concordance & non-discordance principle.
All MCDM techniques have a wide acceptance in decision problems. But in some cases, results could be misleading because of the inaccurate data usage. In general, most of the MCDM techniques evaluate different alternatives according to their importance which is called weights in MCDM terminology, along with their rating/ranking figures. Input data for these assessments are mostly gathered from basic preference, rating a/o ranking questions among alternatives. As the number of questions and alternatives in the questionnaire increases, respondent fatigue may arise. It is a well-known phenomenon that occurs when survey participants become tired of the survey task and the quality of the data they provide begins to deteriorate [5]. One way to avoid this problem is to reduce the total number of alternatives to be evaluated or to change the method of data collection. Conjoint analysis has been seen as the appropriate method for collecting data to be used as input for decision techniques as it is based on making trade-offs between different alternatives as a method of data collection and offers the opportunity to evaluate a large number of alternatives more quickly than basic rating/ranking based questionnaires.
This demonstrative study was conducted under the light of given perspective above. In the following sections of this paper, main objective will be discussed and a brief information about CA, CBC and ELECTRE II will be provided. It will be followed by the results obtained from CBC and ELECTRE II methodologies. The last two sections include discussions and conclusions of the study.

Objectives
The main aim of this study, which has an illustrative perspective, is to demonstrate the usage of conjoint data in ELECTRE II in the context of decision-making process. For this purpose, the university preference problem dealt by both public and private university candidates in Turkey, was chosen as a multi criteria decision problem. The main motif upon the selection of this problem is the rising uncertainty in the recent years among students due to the increase in the number of private universities. Turkish educational system, based on the state universities until 2000's, witnessed a significant rise in the number of private universities, starting from 2010's. The candidates experience confusion about whether their expectations will be fulfilled by these new universities; for there is not enough experience-based data: All the new universities have just started their admissions. Thus, the preference process of the potential students is solely based on their university exam grades. Within this context, the study first clarified the reasons of students' preferences along their importance level by using CA; then used the performance comparisons of the newly established universities' according to these requirements as weights in ELECTRE II and related them to the investment expenditures of these universities.
Detailed information will be provided in the following sections.

Materials and Methods
The application of the proposed approach was made within the framework of the investment expenditures of the universities with the factors affecting the preferences of the university candidates.

Sample
In the context of the purpose, it is important to define targeted population. According to ÖSYM (Student Selection and Placement Center) data, the number of students who attended YGS (Higher Education Entrance Exam) in 2016 and received at least 150 points (minimum score for candidates to make any preference) from any score type was 1.879.812 [6]. This number constitutes the sampling frame of the research with the candidate students. The scope of the study is restricted to private universities located in Istanbul.
Determination of adequate sample size for CA may vary according to the subject of the research and the type of analysis. Akaah and Korgaonkar [7] stated that the sample size was ideal between 100 and 1000, but the observation of 300 and 550 people would be sufficient, while Green and DeSarbo [8] mentioned that they can make effective estimates with smaller samples. In this context, the sample size of the study was determined as 300.
In the sampling process, the private universities in Istanbul, which were considered within the scope of the research objectives, were ranked according to their base scores for university entrance exam and annual tuition fees for the same departments (BA level, departments of administrative sciences). Following the grouping of universities with similar scores and similar fees for the same departments, 7 of 12 newly established universitiesunknown to students-have been randomly selected for sampling. Due to commercial ethics, the names of the selected schools will not be given explicitly within the scope of the study and will be referred to with codes where necessary.
As a result of sampling, two foundation universities (Private universities established by Foundations) were selected for performing the field study. The participants were selected randomly from the candidate students who came to make university choices during the pre-event days of these two universities during the 2016-2017 period.
In addition to collecting preference data via CA, it is planned to collect data from private university administrations in order to determine investment expenditure areas in the aim of being preferable by candidates. This data will be used as attribute weights in the ELECTRE II.

Pilot Study
The first step for the data collection phase of the research is the validity of the reasons of university preference for candidate students by a pilot study. In October 2015, this pilot study was conducted among first grade freshman students in three private universities in Istanbul. These three schools were found to be equivalent to each other in terms of placement scores and prices. Within the scope of the study, 40 freshmen selected by random sampling were interviewed face-to-face. The questioned reasons for preference in the pilot study are mostly taken from previous researches [9][10][11][12][13][14][15][16][17][18] conducted abroad. It is important to assess whether the same reasons are valid in Turkey's process. In this context, an open-ended "Other" option was also added in questionnaire in order to be able to determine an uncovered/exceptional reason by the pilot study. However, it was observed that there were no "other" reasons other than the factors that were previously determined. Hence the pre-defined factors were accepted as valid factors for university preference which will be included in conjoint design.

Conjoint Study
It was thought that timing for Conjoint data collection was really important in terms of taking more accurate answers from the candidates who were just in the university preference stage. For this reason, the optimal period for field study was determined as after the results of both university entrance exams (YGS and LYS) which were conducted in 2016, were announced. In July 2016, computer assisted face-to-face questionnaire was applied and data was collected with 310 people selected from university candidates who came to get information to two universities determined in sampling context. Sawtooth Software 4.10 CBC System 2.7 Module was used in both data collection and analysis phase. In the program, creation of query cards is completely random, so that each interviewer chose from a differentiated set of cards, and all possible combinations were evaluated. The purpose of the study was explained to the participants at the beginning of the questionnaire, and at the moment (during the university preference stage) they were asked whether they would be able to select the options that were shown on the cards.
In the field study, a total of 310 students have been evaluated by computer assisted face-to-face interviews. 14 Questionnaires have been omitted due to incompleteness and a final data of 296 people was included in the analysis.

Weight Data
The information obtained from the university administrations were provided from public relations managers of ten different private universities via e-mail and phone calls between June and August 2016.

Data Analysis
The methodology followed in the data analysis can be explained as follows: 1. As a result of the pilot study conducted in October 2015, the most prominent reasons were used as input in the conjoint design. These preference factors were confirmed both by university administrations and expert guidance teachers, on whether they were significantly important in the process of university choice. 2. Conjoint design was established on the reasons of preference (Academic reputation, Campus facilities, Location / Ease of transportation and department diversity) and the names of the preferred universities. The data collection process was completed in July 2016 with the CBC method based on the prepared design. 3. During the sampling period, the public relations managers of 12 universities have been reached by e-mail and telephone and have been asked about the extent of investment expenditures of their universities on the factors selected within the context of the pilot study and included within the conjoint design (Academic reputation, Campus facilities, Location / Ease of transportation and department diversity). 10 of 12 Universities have provided a response. In order to ease of calculations, the managers were asked to carry out the evaluations of these properties by weighting them over 100 (sum up to 100). 4. The share of the investments made by universities to be preferred was determined as weights and the utility scores of the students' evaluations which had been obtained from CA were used as inputs in the decision matrix in ELECTRE II. As a result of the application of the method, a preference ranking was made among the universities. Conjoint data was analyzed by Sawtooth Software 4.10 CBC System 2.7 package program. Calculations for ELECTRE II were realized with MS Excel 2003.

Methods
A brief information about the methods used will be discussed under this topic.

Conjoint Analysis
Conjoint Analysis, which is defined as a kind of "Thought Experiment" [19], is basically a technique for measuring how and by what a multi-product choice is made by consumers. Therefore, in today's markets where a wide range of products and services are available, new product development or improvement of existing products is one of the main purposes of conjoint methodology in marketing research.
The technique was developed on the basis of Conjoint Measurement by Luce and Tukey [2]. After the introduction in Green and Srinivasan's [20] study, many computer software such as Sawtooth Software, were developed for conducting the analysis. While most of these programs calculate the importance of product features interacting with each other, they also allow the user to simulate different scenarios. Sawtooth Software differs from the others in the design of data collection process and the display of product cards that will enable participants to evaluate the field survey.
As a general definition, CA is a technique that is used in the measurement of preferences and is based on a decompositional approach. In compositional methods, parameters are determined directly by the decision-maker. On the other hand, in decompositional approaches, these parameters are obtained from the holistic evaluation of the answers of the decision makers (rating, preference, purchasing tendency, i.e.). The statistical process behind the analysis works on calculating the contribution of each attribute and its level to the formation of the profiles. According to Green and Srinivasan [20], the basic steps of applying the analysis are: 1. Determination of attributes and levels, 2. Definition of stimulus (product set), 3. Determination of presentation method of stimulus, 4. Determination of the data collection method and measurement scale, 5. Selection of model and estimation method, 6. Selection of simulation element. Decision methods of related steps will be detailed in the next section with their applications.

Determination of Attributes and Levels
The key determinant for deciding which attributes and related levels to include in the conjoint study is associated with the product/service and it is determined by mainly the 14 Using Choice-Based Conjoint Data as Input in ELECTRE II: University Preference Case in Turkey managers of the product/service. Therefore, those who do not have any relative importance in feature selection are not included in the study [21].
Providing the evaluation of all possible features and levels of related product/service by the participant is called as orthogonal factorial design in conjoint studies [22]. This is the most commonly used approach. If four two-level attributes are to be used, the full factorial design will require a total of 2 = 2 4 = 16 product profiles to be evaluated. However, in such designs, the number of profiles to be evaluated by the participants can sometimes be far from providing a healthy assessment. Orthogonal Fractional Factorial Design is often used in such cases [20]. This method allows estimation of the utility scores (a.k.a. main effects) of the attributes. On the other hand, these designs can predict more limited (e.g. first level) interactional effects. For instance, if there is a full profile design in which five two-level attributes will be evaluated (32 profiles in total), using a half-replicate design approach can reduce the number of profiles to be evaluated from 32 to 16. In such a design, some of the main and interactive effects can be confused. Although this method is used in some applications, the number of profiles to be evaluated remains high, so some profiles are combined into blocks to form a partially balanced block design. In practice, participants evaluate different blocks of profiles.

Definition of Stimulus
Once the properties and levels of the product have been determined, the product combination to be evaluated is called as stimulus. As an example, given is an ice-cream product with three attributes which are flavor, scoop and topping. Related levels of each attribute are chocolate and vanilla, double and single, no toppings and chocolate chunks respectively. An example stimulus set of mentioned case is illustrated below.

. Determination of Presentation Method for Stimulus
After the formation of product profiles, it should be decided how to display these options to the participant.
Conjoint studies generally use the following methods [23]: 1. Verbal identification: A text that describes the product profile verbally is shown 2. Profile cards: Show product profile cards containing descriptions of each attribute and levels 3. Visual identification: To show the drawing or photograph of the product to be evaluated 4. Computer-aided design: It is possible to show the product designs that don't exist 5. Physical evaluation: The display of the product itself After the creation of product profiles, the participant should decide how to display these options. Conjoint studies generally use the following methods:

Determination of the Data Collection Method and Measurement Scale
Conjoint Techniques differ in terms of types of studies in marketing research. All of the approaches, which can basically be grouped under three headings, have specific advantages and disadvantages. Choosing the appropriate method for the purpose of research is one of the important points to be decided. The approaches mentioned are [21]: 1. Adaptive Conjoint Analysis: The method developed by Sawtooth Software is preferred in the aim of having a user-friendly interface. However, in some cases it is insufficient to give optimal results. The greatest advantage of this method is that the full profile method can be used to implement the application without evaluating all product features simultaneously. However, with this approach, more than six attributes cannot be evaluated effectively at the same time. Yet this method is also effective in estimating price sensitivity. 2. Conjoint Value Analysis: This method can be used to determine the interactions between utility values for each attribute by using a full profile, pairwise comparison or trade-off approaches. 3. Choice-Based Conjoint Analysis: CBC is the most up-to-date conjoint method and allows the calculation of common interactions in product characteristics. In this method, the participant chooses among the product profiles which are defined according to their various characteristics rather than evaluating them with ranking or points. If the purpose of the research is to determine the preference of the product or service of interest, it should be taken into consideration that no product will be preferred as in real life. Because if the participant does not find the features he / she is looking for in any of the defined products, he / she will not choose naturally. Although the researchers argue that each participant has distinct characteristics and that some features cannot be clearly seen unless the answers are analyzed on a personal basis, the paired comparison method in CBC allows the determination of the actual variables which can be ignored in some product features especially in the studies where the product price is included. When scoring or ranking are performed, these features may not fully demonstrate their importance. Another advantage of this method is that the calculations are relatively short and easy. Due to the complexity of the conjoint studies, the most appropriate way to collect data is face-to-face interviews. In this regard, data collection by phone or mail is not preferred due to the possibility of making the product profiles difficult to understand. On the other hand, performing computer-assisted interviews, rather than pen-and-paper alternative, is better especially for adaptive and choice-based conjoint methods [24]. The other data collection methods are detailed as follows: 1. one of the profile sets where each profile is determined by all properties In both the full profile and trade-off approach, evaluation can be based on ranking or scoring. Scores are usually made on an evenly spaced scale of 7 or 9. The paired comparison approach is used in the Adaptive Conjoint Analysis method.

Selection Model and Estimation Method
There are four basic models describing the functions that express the multiple attribute preferences of consumers in the context of Conjoint. These are vector model, ideal point model, part-worth model and mixed model. The mathematical representation of these four models is as follows: 1. Vector Model: Many different methods have been used to evaluate the conjoint data. The selection of the appropriate method is based on the scale that the participant uses in his / her preference assessment for product profiles. The three basic scales used in the conjoint are: metric / ratio, ranking and nominal (selection based) scales. Metric data is generally used for prediction with ordinary least squares and its significance levels are obtained from here. The most commonly used methods for measurement based on order MONANOVA [25] and LINMAP [26]. Another alternative is the PROBIT model and does not require the assumptions of the LOGIT [27]. However, in the estimation of this model, when the set subject to the selection contains more than three alternatives, it may be problematic that the numerical solution of the fourth and higher dimensional integrals cannot be made. In such cases, McFadden [28] proposed the use of simulation methods for integral calculation.
Sawtooth Software, R and SPSS are easy-to-use programs to estimate the results. For selection-based, adaptive and value-analysis approaches Sawtooth simulation model provides more detailed results than SPSS [29].
However, in conjoint studies, it is very important to collect the data correctly and accurately before the analysis. Although design can be done with many statistical analysis programs such as SPSS Conjoint, R's optfederov () function in AlgDesign package. Sawtooth Software provides a great advantage with its interface defined in the process. Otherwise, these product cards to be evaluated should be prepared one by one by evaluating each participant with the appropriate combinations.

ELECTRE
The technique was proposed by Roy [30] in 1968 by introducing different techniques with the work of Roy, Bouyssou and Yu, Roy and Bertier [4,31,32]. ELECTRE has six different approaches which are called ELECTRE I, ELECTRE II, ELECTRE III, ELECTRE IV, ELECTRE IS, and ELECTRE Tri. All these versions are based on the same fundamental concepts but they differ on some operations and on the type of the decision problem. Specifically, ELECTRE I is used for selecting the problems, ELECTRE TRI for the assignment of those problems and ELECTRE II, III and IV for ranking of the problems [33].
ELECTRE compares all possible pairs of different alternatives on the basis of criteria and takes decisions based on criterion-based scores (Hwang and Yoon, 1981). In binary comparisons, alternatives are chosen by selecting superior performance alternatives. In the process of the method, a decision (pay-off) matrix is created where the performance of all the criteria is shown at the level of available alternatives. The lines in the matrix show alternatives, columns indicate the criteria, the matrix elements indicate the performance of the benchmark alternative in the corresponding row and column.
Then normalization process is performed similar to Analytical Hierarchy Process (AHP). The square root of the sum of the squares of the column elements is used here. The In next step, the normalized decision matrix is multiplied by the benchmark weights (0 ≤ ≤ 1; ∑ =1 ) obtained as a priori information to obtain the weighted decision matrix (V), shown below.
( − 1) number of concordance sets are created for each alternative pair using the weighted decision matrix (V) to determine concordance and discordance matrix. Here, the number of elements must be equal to the number of criteria.
Concordance sets (for instance, Alternative #1 and Alternative #2) can be shown as follows: The formula is basically based on comparing row elements with each other in sequence. For instance, when comparing Alternative #1 and Alternative #2, the value of the Alternative #1 for the criteria in third and fourth columns will be equal to or greater than that of the Alternative #2. If it is smaller for other columns, the value will be 12 = {3,4}.
Discordance sets are also mutually generated on each set of concordance. The path to be followed in the creation of the discordance matrix is likewise based on the comparison of the line elements. Again, for Alternative #1 and Alternative #2, a set of discordance can be created as follows: Discordance set against 12 in the example above (for a four criterion example) will be 12 = {1,2}. The concordance matrix C and the discordance matrix D are generated using concordance and discordance sets. The approach to constructing ( × ) dimensional compatibility matrix will be used to determine the relative weights for the criteria where the alternatives are superior (or equal). The elements of the concordance matrix that do not value for the diagonal elements are created as follows: If 12 = {3,4}, according to the above example, the value of the element 12 of matrix C will be calculated as 12 = 1 + 2 . Elements of the discordance matrix are obtained using the following formulation: In the formulae, numerator shows the maximum difference on discordance index and denominator shows the maximum difference on any creation. The D matrix is also ( × )dimensional like C and does not take any value for its diagonal elements.
In the next step towards determining the dominant criterion, concordance and discordance indexes are made to calculate the matrices of superiority. The concordance index for a pair of alternatives #1 and #2, measures the strength of the hypothesis that alternative #1 is at least as good as alternative #2. The discordance index measures the strength of evidence against this hypothesis. Final ranking of alternatives is obtained from the superiorities according to these indexes. The concordance superiority matrix (F) is ( × ) dimensional in size and is found by comparing the alignment matrix elements with the concordance threshold values. If the concordance matrix element is equal to or greater than the threshold value, the element of the concordance superiority matrix takes 1, otherwise takes 0.
The matrix elements take only 1 and 0 values. The following formula is used to calculate threshold values: Discordance superiority matrix (G) elements are similarly calculated by comparing discordance matrix elements with the threshold value of discordance. If the discordance matrix element is equal to or greater than the threshold value, the element of the discordance superiority matrix takes 1, otherwise takes 0.
The elements of the ( × ) dimensional discordance superiority matrix can also take only 1 and 0 values. Calculation of the threshold values is similar to the other and uses the following formula: In ELECTRE II, it was necessary to increase threshold values from the average concordance and discordance threshold values in order to evaluate different situations [34].
The total superiority matrix (E) to be used in decision-making is obtained by comparing the elements of concordance superiority and discordance superiority matrix in order to evaluate outranking relations. The lines and columns of the E matrix represent alternatives, and the elements with 1 in both matrices are represented as 1 in E matrix, and 0 in any of them. Hence, it is decided by comparing superior alternatives.
Two elements are used to establish the outranking relationship value for ELECTRE II which are strong ( )and weak outranking relationship( ). According to the definition of and, decision makers must determine different concordance index levels and discordance index reducible level. It is assumed that, * , 0 , − are represented by three degressive concordance levels, respectively, and meet the following condition: 0 ≤ − ≤ 0 ≤ * ≤ 1 Plus, * , 0 are represented by two degressive discordance levels, and meet the following conditions: 0 < * < 0 < 1 Posterior to identify concordance and discordance levels, it is possible to establish value for the and, , calculate strong ( ′ ) and weak ( ′′ ) ranking values, and finally, determine the average ranking � for final ranking result [35].

Results
In this chapter, the findings of the pilot study, the results of the conjoint analysis which was based on the preferences of the university candidates and ELECTRE II integration practice realized with conjoint measurement data are represented.

Pilot Study
In the face-to-face pilot survey study, which was attended by 40 freshmen, the reasons of university preference in the literature were offered to the students for their evaluation.
Students were asked to mark the factors they considered important in university preferences, and then they were asked to rank 1 to 3 by selecting the three factors they considered the most important. According to the pilot study, the factors that are important in the university preference of the students are as follows:

Conjoint Study
According to the results obtained from the pilot study, the following five factors with 16 levels in total, will be included in stimuli to be prepared for the evaluation of candidate students within the scope of conjoint design: The effectiveness of the design was examined before the application of the conjoint analysis. The procedure used to investigate the effectiveness of the design is found in the Sawtooth Software CBC module. This method is intended to investigate the effectiveness of the design to be tested rather than the choice of respondents. Based on the least squares estimation procedure, the effectiveness of each factor's coefficient of estimation is examined. Thus, it allows to conduct a comparative effectiveness study at the level of the sub-attributes of each factor [36]. The emphasis in this test is the pattern of relative quantities analogous to each other, not a definite estimate of each standard error for a certain number of participants. In estimation procedure, first level of each attribute is removed. Within this context, the first level of each factor is automatically deleted from this analysis. The column "Actual value" gives the estimated standard error for the analyzed data. The column named "Ideal value" gives an estimate of what these standard errors will be if the design is fully orthogonal and has the same number of observations. In the analysis output given in Table 3, the description of the actual and ideal values given for each level is as follows: The ideal value calculated for the relevant level is independent of the respondents, with the specified sample size and with the optimal value for the corresponding level in an orthogonal design framework, while the actual value is calculated with the data obtained. The minimum difference between them is considered to be the main indicator of the effectiveness of the design. The "Efficiency" column, compared to the hypothetical orthogonal design (which is the square of these ratios), gives the relative efficiency of this design for estimation of each parameter. When the "Efficiency" column in Table 3 is examined, it appears that the random design has an average activity of about 99 percent relative to a hypothetical orthogonal design. This value is an indicator of the effectiveness of the design despite the hypothetical orthogonal design. The median efficiency measure of a randomized design is 0.97. The values can be interpreted accordingly [36].   In the evaluation of the predictive power and adequacy of the model, it is accepted that larger values are better for criteria calculated above. Therefore, predicted model was also found effective [37]. The results obtained from Multinomial LOGIT Model were analyzed by the levels of each factor. Detailed results of estimation are shown in Table 4 below. According to results:  Importance degree of "Experienced faculty members" level for "Academic reputation" attribute interpreted as 0.51143. This coefficient is statistically significant ( = 25.5, < 0.05).  Universities that are considered equivalent in the University factor are shown. Among these, University A was found to have the highest level of importance (effect: 0.31059) ( = 5.56, < 0.05).This value is positive and statistically significant.  When the "Presence of the desired field" attribute is taken into consideration, it is seen that "The field I want to study is available" level has the highest importance for candidate students (effect: 0.7972) and the coefficient of utility in the model is statistically significant ( = 38.23, < 0.05). The opposite level of the same attribute has the same but negative significant value.  Among the levels in the "Location / Ease of transportation" attribute, the "Easy to get to & away from home" level has the highest utility. (effect: 0.11018) ( = 3.63, < 0.05). Another remarkable point here is that the reverse effect of "It is difficult to get to & away from home" level is quite high. (effect: -0.1955) ( = −6.83, < 0.05). The third level in this attribute is; "Close to home & easy to reach" has almost the same degree of significance as the "Easy to get to & away from home" level and the coefficient is statistically significant (effect: 0.0936) ( = 3.3, < 0.05). Finally, the coefficient of level "Has an attractive campus" for the "Campus Facilities" attribute is statistically significant.

Main Effects
As a result of the analysis made for the main effects, it was observed that "The field I want to study is available" was more preferred than "The field I want to study is not available" in 77% of the cases in "Presence of the desired field" attribute ( 2 = 1738.38, < 0.01). In the "University" attribute, it was found that A University was preferred more than the other school options with a preference of 55% ( 2 = 13.33, < 0.05). Among the levels of "Academic reputation", the "Experienced faculty members" is the most preferred level with 65% ( 2 = 511.42, < 0.01). The most preferred level for "Campus Facilities" attribute is "Has an attractive campus" with 60% preference percentage ( 2 = 217.99, < 0.01). Among the levels in the "Location / Ease of transportation" attribute, "Easy to get to & away from home" level is the most preferred one with 53% ( 2 = 24.21, < 0.01). The percentages of preference for each factor are given in the Table 5 below.

Determining the Weights of Universities' Marketing Activities
In order to be used as weights in ELECTRE II procedure, the properties that universities invested in have been determined from the official personnel interviewed in the selected schools (public relations authority or general secretary). During the interviews, the distribution of investments in order to increase the level of preference has been asked to the related personnel. These distribution fields, as stated before, have been gathered from the factors that students placed an emphasis on during their preference period. In the aim of presenting the university's point of view, the name of the university was categorized under "Brand Awareness", while the "Presence of the desired field" was categorized under "Department Diversity". The rest was recorded as per se.
In this context, they were asked to distribute the total 100 points given to the five factors presented to them in a way that represents the relative and individual importance level. Correspondingly, the evaluations made by the school authorities are given in Table 6 below.
According to this, it is evident that the school administrations have invested most in increasing their academic reputation by hiring qualified faculty members, and this factor is followed by investing on brand awareness activities for their schools. The least invested factor is the location of the school and the ease of transportation to school.
Proportional distribution of investments for each factor in a 100-sum scale was organized and the arithmetic mean of attribute based data was accepted as the weighting vector in ELECTRE II.
In the following chapter, ELECTRE II application results are detailed and corresponding findings are evaluated in the context of university preference problem.

ELECTRE II
This section depicts the application steps of ELECTRE II method separately. The results of the calculations explained in the Methodology Section in detail are exhibited in matrixes; while their explanations are placed in-between the tables from 7 to 11.
Initial decision (pay-off) matrix (A) of ELECTRE is constituted from conjoint measurement results showed in Table  5. In this matrix where cell values are the performance value of the alternative on related criterion showed below: Normalized decision matrix is calculated by using Equation (5) discussed before. The purpose of normalization is to obtain dimensionless values of different criteria to make them comparable with each other. Weighted decision matrix (V) is calculated as multiplying the normalized decision matrix with the weight vector. The investments that the universities provided for the preference factors are used as weight values in this part. The distribution of these investment expenditures were already gathered from the school administrations by the division of 100 points according to the weight of the factors. Here, weight values are indexed to 1 for ease of calculation. These indexed values constitute the Indexed Weight Vector showed below. The concordance index measures a weighted number of attributes for which alternative x is preferred or equal to alternative y. In this study these alternatives indicate the evaluated universities. Concordance values were calculated by using Equation (7) and presented in Table 10 below. The discordance index. measures the strength of the greatest discontent or disagreement among all attributes. When alternative x is selected over alternative y. In other words, it indicates the level of discontent that is accepted when attribute x is selected instead of attribute y. Calculation steps of discordance values were described in Equation (11) before and results presented in Table 11 below. After computing the concordance and discordance indices for each pair of alternatives, two types of outranking relations are built by comparing these indices with two pairs of threshold values which are: * , * , − , − . * , indicates the average value of ( , ) . * , indicates the average value of ( , ). − and − are defined by decision maker accordingly, * > − , and * < − . In the final interpretation phase of outranking relations two main rule applied [35].  Table 12.
Final rankings of universities which constituted from superiority matrix, could be presented as:

Conclusions and Discussion
ELECTRE II is a MCDM technique that provides rankings according to superiorities of different alternatives based on to their attributes' performance scores. Evaluation method of the technique is based on pairwise comparison of alternatives by concordance & discordance principle. However, use of ELECTRE, as well as most MCDM techniques, needs weight values, which are mostly determined subjectively by the decision maker in application. The context of analysis includes the evaluation of various alternatives uniquely within themselves based on different factors rather than a comparison between these various alternatives. The data obtained from this single dimensional evaluation affects the practicality of the results. In order to eliminate this negative tendency and deriving more objective indicators, CA was used to gather alternative based evaluation data. This approach was applied by Zardari, Cordery and Sharma [34] to determine priority in water allocations by using standard conjoint design. But in this study, an eminent approach in conjoint measurement, Choice-Based Conjoint was adopted to gather preference data.
As an application of the proposed approach, the university preference problem for candidates in Turkey, was chosen as a multi criteria decision problem. The main reason for this selection lies on the fact that state universities have been ruled by Turkey's higher education system for years. Recently, the number of private universities arose and they started to take role in university candidates' selection paradigm. However, as sufficient information cannot be gathered about these schools recently included in the system, there appears a significant uncertainty during the preference process. Although the university candidates make their preferences on the assumption that their requirements will be met by these newly established schools, as stated before, there is no concrete evidence that these institutions will fulfill such factors, given that they have not graduated any students yet.
In this context, it is necessary to obtain an objective ranking among these private universities by considering important factors according to the candidates' university selection process. Thus, interpreting the preference scores of the candidates based solely on CA would not be sufficient. The schools' performances based on the selected factors by the students would also be critical in the objective ranking.
Stated multi criteria decision problem fits with the main objective of this demonstrative study, which aims to present usage of conjoint data in ELECTRE II in the context of decision-making process. By application of the stated approach, it is gathered an objective ranking among substitute private universities. ELECTRE II procedure is used with evaluations of factors affecting the private university's preference among candidates and data weighted by investments of the universities correspond to the performances of the universities in each factor. Preference data were collected by CBC method from 296 students who were in the preference process after 2016 university entrance exams. According to CBC results, some of the most important factors in preference process were appeared as, "presence of the desired field", "academic reputation of university" and "campus facilities" respectively. Conjoint scores of these factors were used to develop payoff matrix (universities vs. preference factors array) in ELECTRE II. In order to gain weights of each attribute, data was collected from administrative level directors of selected private universities. Proportional distribution of investments for each factor in a 100-sum scale was obtained from in-phone interviews or by e-mail and factor-based indexed average of the collected data accepted as the weighting vector. The results obtained from both CBC and weights were used as input in ELECTRE II in order to determine a complete and objective ranking of universities. By using proposed integrated approach, solution to a university selection problem in the field of education was investigated.
As a result of the study, realized by empirical data, it could be seen how the rankings differ according to university candidate preferences when investment decision according to different areas of universities change the weights of attributes defined as preference reasons. In addition to that, this approach also allowed to describe the market situation in general thus each university could make a comparative assessment of its own.
Another advantage of the offered approach is that, given the input data in the ELECTRE method, it provides a more objective evaluation within the nature of the problem compared to the subjective evaluation of the decision maker. In numerous ELECTRE implications, the weights are specified by a specific decision maker or a limited number of experts on the subject. Yet, the data about the performances of different alternatives within the analysis is obtained by the singular evaluation of each participant. Thus, the approach of this study has increased the quality and coherence of the data.
Future studies may use the CA data with different MCDM methods, as well as implicate the same approach on the decision making problems in wider and/or different fields.